Foreman-maintain health check fails

Support
1

Problem:
foreman-maintain health check
Check whether all services are running using the ping call: [FAIL]
Couldn’t connect to the server: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

Expected outcome:
Check whether all services are running using the ping call: [OK]

Foreman and Proxy versions:
foreman-3.4.1-1.el8.noarch
katello-4.6.0-1.el8.noarch

Distribution and version:
RHEL8
Other relevant data:

The problem arised after upgrade from 3.3/4.5 to 3.4/4.6. Could not find any hints in the logs regarding this error. So Ive no idea which certificate does not pass the validation.

root@foreman-test [Foreman-Test-Server] /etc/pki/ca-trust/source/anchors # openssl s_client -connect foreman-test.example.com:443
CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4
verify return:1
depth=0 C = AT, ST = **City**, O = "company", CN = *.example.com
verify return:1
---
Certificate chain
 0 s:C = AT, ST = **City**, O = "company", CN = *.example.com
   i:C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4
 1 s:C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4
   i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
   i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
snip
-----END CERTIFICATE-----
subject=C = AT, ST = **City**, O = "company", CN = *.example.com

issuer=C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4

---
Acceptable client certificate CA names
C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = SomeOrgUnit, CN = foreman-test.example.com
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 6226 bytes and written 431 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 3072 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)


root@foreman-test [Foreman-Test-Server] /etc/pki/ca-trust/source/anchors # hammer ping
database:
    Status:          ok
    Server Response: Duration: 0ms
candlepin:
    Status:          ok
    Server Response: Duration: 31ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 26ms
candlepin_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:
    Status:          ok
    Server Response: Duration: 83ms
pulp3_content:
    Status:          ok
    Server Response: Duration: 126ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 2ms

Does anybody know how to debug this issue?

2

It seems to be a bug to me. I get the same:

[root@foreman8 ~]#  foreman-maintain health check
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [FAIL]
Couldn't connect to the server: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)] no
Check for paused tasks:                                               [OK]      
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario::FilteredScenario] failed.

The following steps ended up in failing state:

  [server-ping]

Resolve the failed steps and rerun the command.
In case the failures are false positives, use
--whitelist="server-ping"

If I didn’t miss something, it tries to verify against /etc/pki/katello/certs/katello-default-ca.crt and obviously fails. This is the openssl command:

[root@foreman8 ~]# openssl s_client --CAfile /etc/pki/katello/certs/katello-default-ca.crt -connect foreman8.example.com:443
...
    Verify return code: 19 (self signed certificate in certificate chain)

Instead it should use /etc/pki/katello/certs/katello-server-ca.crt.

Maybe @evgeni or one of the other developers can shed some light on it?

3

ah ok :slight_smile: so the health itself is buggy? Can you explain how you found that the health check verifies against the /etc/pki/katello/certs/katello-default-ca.crt

So the solution in case foreman-maintain is used in monitoring should be:
foreman-maintain health check --whitelist="server-ping" for the moment I guess.

thank you for quick reply!

4

Yes. I think so. Unless you have manually made changes to the configuration but only used foreman-installer, a ssl verification error shouldn’t happen. If you fix the ssl verification error the check shows OK.

I have used strace on the command to trace the system calls. In the process, it opens katello-default-ca.crt but not katello-server-ca.crt which would contain the ca chain for https port 443.

# strace foreman-maintain health check --label server-ping 2>&1 | grep ^open

As it’s in the end after opening openssl.config, it is pretty safe to assume that it’s used to establish and verify the connection. If you like you can temporarily add the server-ca to the default-ca and you’ll see that the command it working:

[root@foreman8 ~]# cd /etc/pki/katello/certs/
[root@foreman8 certs]# mv katello-default-ca.crt katello-default-ca.crt.ORIG
[root@foreman8 certs]# cat > katello-default-ca.crt katello-default-ca.crt.ORIG katello-server-ca.crt
[root@foreman8 certs]# foreman-maintain health check --label server-ping 
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------

[root@foreman8 certs]# /bin/mv katello-default-ca.crt.ORIG katello-default-ca.crt

Thus, if you add server-ca to default-ca it’s O.K. (for security reasons, don’t keep it in there) default-ca is the CA for internal connections and server-ca for the connection to public connections to port 443. So it’s either using a different file or the API connection worked differently before.

5

Ah ok makes sense thank you!

I will keep everything as it is since everything works as exepcted and gonna change only the icinga-check accordingly skipping the ping-call. Hope that this will be fixed in next release(s)

Thank you for your support!

6

You’re correct, this seems to be a regression in Fixes #35058 - upgrade scenario for Foreman and Katello nightly (#633) · theforeman/foreman_maintain@b0337ee · GitHub

I’ve opened Bug #35892: health check uses the wrong certificate bundle to talk to Foreman - Foreman Maintain - Foreman

7

Would you mind checking the patch in Fixes #35892 - use server_ssl_chain to check certs by evgeni · Pull Request #668 · theforeman/foreman_maintain · GitHub?

8

patch obviously works for the ping-call. In the process, it opens katello-server-ca.crt.

openat(AT_FDCWD, "/etc/crypto-policies/back-ends/openssl.config", O_RDONLY) = 7
openat(AT_FDCWD, "/etc/pki/katello/certs/katello-server-ca.crt", O_RDONLY) = 7
openat(AT_FDCWD, "/var/lib/foreman-maintain/data.yml", O_RDONLY|O_CLOEXEC) = 6

Thank you!

9

Perfect, this got merged and I’ll see that we release it soon.

Thanks for noticing, reporting and testing!

10

The patch works for me, too:

[root@foreman8 ~]# foreman-maintain health check --label server-ping 
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
11

This would be handy if pushed to the most current 3.4 release :slight_smile:

12

We’re working on that :wink:

Starting in nightly with Update ruby-foreman-maintain to 1.2.2 by github-actions[bot] · Pull Request #8873 · theforeman/foreman-packaging · GitHub and Update rubygem-foreman_maintain to 1.2.2 by github-actions[bot] · Pull Request #8876 · theforeman/foreman-packaging · GitHub

1 Like
13

Idid the manual PR fix for maintain for now. I was finally able to upgrade to 3.5 successfully as well. :slight_smile: Thanks for all the support like always from these forums!

14

Also confirming this works great on 3.5.1 (had this issue since deploying 3.5.0-rc2) after patching rubygem-foreman_maintain 1.2.1.

15

I’ve prepared picks for both Debian and EL setups, 3.4 and 3.5:

Should go out to the repos in the next dot release (but I have no idea when that is planned)

16

Foreman 3.5.1 and katello 4.6.0 still have the same issue. Updating rubygem-foreman_maintain to the latest nightly version (1.2.4) addressed the issue. https://yum.theforeman.org/releases/nightly/el8/x86_64/rubygem-foreman_maintain-1.2.4-1.el8.noarch.rpm

17

The fix was introduced in 1.2.2 but nothing past 1.2.1 has been built and shipped in stable yet, unfortunately.

https://yum.theforeman.org/releases/3.5/el8/x86_64/

I have the patch applied manually, but I’m hoping for the fix to be generally released, maybe in step with 3.6.0?

18

3.5.2 is around the corner and will contain said fix.