Foreman 3.11/Katello4.13 - Smartproxy install/update issues with custom CA

[lakedh]

Problem:
Foreman 3.11 installed properly on the main server and functioned well. While updating the smart proxies connected to them the installer fails with ssl errors
Expected outcome:
insttaller should run
Foreman and Proxy versions:
3.11
Foreman and Proxy plugin versions:

Distribution and version:
RHEL 8.9
Other relevant data:

rechecking with kattello-cert-check and reinstalling the certificates on the mainserver does not report any errors. Trying to install a freshly created cert bundle on the smartproxies fails with same error.

Cert chain provided by main server itself is looking fine, its providing the server, issuing and root cert.

Issue seems to be not present prior to the update. There foreman-installer can be run without an error on the smart proxies.

2024-07-04 13:16:51 [NOTICE] [pre] Migrating PostgreSQL data
2024-07-04 13:16:59 [NOTICE] [pre] Analyzing the new PostgreSQL cluster
2024-07-04 13:17:22 [NOTICE] [pre] Upgrade to PostgreSQL 13 completed
2024-07-04 13:17:25 [NOTICE] [configure] Starting system configuration.
2024-07-04 13:17:38 [NOTICE] [configure] 250 configuration steps out of 1194 steps complete.
2024-07-04 13:17:41 [NOTICE] [configure] 500 configuration steps out of 1196 steps complete.
2024-07-04 13:17:42 [NOTICE] [configure] 750 configuration steps out of 1200 steps complete.
2024-07-04 13:17:44 [NOTICE] [configure] 1000 configuration steps out of 1221 steps complete.
2024-07-04 13:18:22 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-smartproxy1.domain.tld]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/hosts?search=name%3D%22smartproxy1.domain.tld%22
2024-07-04 13:18:22 [ERROR ] [configure] Wrapped exception:
2024-07-04 13:18:22 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-07-04 13:18:22 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smartproxy1.domain.tld]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22
2024-07-04 13:18:22 [ERROR ] [configure] Wrapped exception:
2024-07-04 13:18:22 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-07-04 13:18:22 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smartproxy1.domain.tld]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22
2024-07-04 13:18:22 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smartproxy1.domain.tld]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22
2024-07-04 13:18:22 [ERROR ] [configure] Wrapped exception:
2024-07-04 13:18:22 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-07-04 13:18:26 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Foreman_host resource 'foreman-proxy-smartproxy1.domain.tld' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-smartproxy1.domain.tld]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1209 of 1224)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/hosts?search=name%3D%22smartproxy1.domain.tld%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_host[foreman-proxy-smartproxy1.domain.tld](provider=rest_v3)
    Making get request to https://mainserver.domain.tld/api/v2/hosts?search=name%3D%22smartproxy1.domain.tld%22
Error 2: Puppet Foreman_smartproxy resource 'smartproxy1.domain.tld' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smartproxy1.domain.tld]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1211 of 1224)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22
    Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.02 seconds
  Foreman_smartproxy[smartproxy1.domain.tld](provider=rest_v3)
    Making get request to https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22
    Making get request to https://mainserver.domain.tld/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22

[lakedh]

Upgrade was done from 3.10 to 3.11. As issue is not present within the 3.10 I suspect it might be introduced during the update. Of course it can also be a local only issue with the certificates. Any idea on what I should focus on?

[selfsealingstembolts]

I’ve also encountered this error trying to upgrade a smart proxy to Foreman 3.11/Katello 4.13. The smart proxy in question was installed with

foreman-installer\                                         
                    --scenario foreman-proxy-content\
                    --certs-tar-file                              "/root/smart-proxy.example.com-certs.tar"\
                    --foreman-proxy-register-in-foreman           "true"\
                    --foreman-proxy-foreman-base-url              "https://katello.example.com"\
                    --foreman-proxy-trusted-hosts                 "katello.example.com"\
                    --foreman-proxy-trusted-hosts                 "smart-proxy.example.com"\
                    --foreman-proxy-oauth-consumer-key            "***"\
                    --foreman-proxy-oauth-consumer-secret         "***"

Running this same command, or just foreman-installer, results in the same SSL errors as OP. The custom certs tarball is still in place on the smart proxy; deleting the extracted /root/ssl-build folder and allowing the installer to extract the contents of the cert tarball results in identical behavior.

I’ve narrowed the issue down to foreman-installer overwriting /etc/foreman-proxy/foreman_ssl_ca.pem with the contents of /root/ssl-build/katello-server-ca.crt, which should be the custom CA used by Katello. Indeed manually extracting the certs tarball and examining its contents shows the correct katello-server-ca.crt. However, the foreman-installer inflated contents of /root/ssl-build/katello-server-ca.crt is incorrect; the file is identical to /root/ssl-build/katello-default-ca.crt, which is not the custom CA Katello is using.

It seems that foreman-installer is incorrectly copying /root/ssl-build/katello-default-ca.crt to /root/ssl-build/katello-server-ca.crt, which is then copied to /etc/foreman-proxy/foreman_ssl_ca.pem.

[w4y_secker]

Hi!
I have the same Problem with upgrading smart proxies from 3.10/4.12 to 3.11/4.13. Foreman Upgrade was fine without problems

[selfsealingstembolts]

I believe I’ve identified the issue. Foreman 3.11 uses v18.0.0 of theforeman/puppet-certs, which introduces a number of changes in certificate handling. Commit 433dadc changes the way an undefined value for certs::server_ca_cert in your /etc/foreman-installer/scenarios.d/foreman-proxy-content-answers.yaml file is treated.

You can see here that these changes will result in the module copying /root/ssl-build/katello-default-ca.crt to /root/ssl-build/katello-server-ca.crt, which is then picked up later on and copied to /etc/pki/katello/certs/katello-server-ca.crt and ultimately /etc/foreman-proxy/foreman_ssl_ca.pem.

A potential workaround is explicitly set the values of certs::server_cert, certs::server_key, and certs::server_ca_cert in /etc/foreman-installer/scenarios.d/foreman-proxy-content-answers.yaml like so:

certs:
  server_cert: "/root/ssl-build/${smart-proxy.example.com}/${smart-proxy.example.com}-foreman-proxy.crt"
  server_key: "/root/ssl-build/${smart-proxy.example.com}/${smart-proxy.example.com}-foreman-proxy.key"
  server_ca_cert: "/root/ssl-build/katello-server-ca.crt"

replacing ${smart-proxy.example.com} with the FQDN of your smart proxy.

The values of certs::server_cert and certs:server_key are required along with certs:server_ca_cert otherwise the installer will fail on the katello-certs-check step.

[w4y_secker]

Thank you!
Workaround is working.

[JendaVodka]

Hi,

will this issue be addressed somehow by the foreman/katello team ? It quite complicates the upgrade to 3.11/4.13 with custom certificates.

Thank you.

[ekohl]

I missed this, but luckily @selfsealingstembolts created an issue that I did see: Custom CA overwritten when certs::server_ca_cert is undefined · Issue #456 · theforeman/puppet-certs · GitHub and Fixes #37817: Only copy server CA in build root if generate is true by ehelms · Pull Request #463 · theforeman/puppet-certs · GitHub was merged.

Here’s at least the PR to update it in 3.12 so in GA that we intend to release next week it’ll be fixed:

Fixes #37817 - Update katello-certs to 19.0.1 for custom certs

Looks like 3.11 needs another module release so I’ll at least set a target version as a reminder.

[gvde]

@ekohl any chance to get this into 3.11? Seems quite a few people are running into problems due to this…

[ekohl]

Release Team Meeting 2024-10-09

tl;dr: 3.11.4. 3.11.3 was tagged (you can see that in git) but it contains exactly the same code as 3.11.2. The team was confused how this happened and figured they’d wait for me to get back from PTO. Everything should be sorted out for 3.11.4 that should include fixes. And I now see I forgot to write it down, but it was discussed that early next week we’ll start the release train.

[ehrenreich]

I have just tried to upgrade on Almalinux 8 running 3.11.4 to 3.12 and it seems the problem exists again on the smartproxies.

Could it be that this does not work as intented?