Puppet certificate error after katello install with custom certificate

Support
, ,
1

Problem:

I installed a Foreman 3.5 Katello 4.7 with custom certificates signed by our internal CA.
At the end of the installation, these errors appeared :

2023-03-20 14:31:27 [NOTICE] [configure] 1750 configuration steps out of 1850 steps complete.
2023-03-20 14:31:58 [ERROR ] [configure] Error making POST request to Foreman at https://foreman.example.com/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://foreman.example.com:9090/v2/features Please check the proxy is configured and running on the host.
2023-03-20 14:31:58 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]/ensure: change from 'absent' to 'present' failed: Error making POST request to Foreman at https://[[foreman.example.com](http://foreman.example.com)](http://foreman.example.com)/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://foreman.example.com:9090/v2/features Please check the proxy is configured and running on the host.
2023-03-20 14:32:02 [NOTICE] [configure] System configuration has finished.

Services are up and running, the GUI seems ok. The Foreman is installed with an integrated puppet server. The installation command :

foreman-installer --scenario katello \
> --certs-server-cert "/root/certs/server_cert.pem" \
> --certs-server-key "/root/certs/server_cert_key.pem" \
> --certs-server-ca-cert "/root/certs/CA_bundle.pem" \
> --enable-foreman-plugin-puppet \
> --enable-foreman-cli-puppet \
> --foreman-proxy-puppet true \
> --foreman-proxy-puppetca true \
> --foreman-proxy-content-puppet true \
> --enable-puppet \
> --puppet-server true \
> --puppet-server-foreman-ssl-ca /etc/pki/katello/puppet/puppet_client_ca.crt \
> --puppet-server-foreman-ssl-cert /etc/pki/katello/puppet/puppet_client.crt \
> --puppet-server-foreman-ssl-key /etc/pki/katello/puppet/puppet_client.key \
> --enable-foreman-plugin-column-view \
> --enable-foreman-plugin-monitoring \
> --foreman-initial-organization 'Test'

Now Iā€™ve got an issue with the puppet agent :

Info: Using environment 'production'
Debug: Resolving service 'fileserver' using Puppet::HTTP::Resolver::Settings
Debug: Creating new connection for https://foreman.example.com:8140
Debug: Starting connection for https://foreman.example.com:8140
Error: certificate verify failed [self signed certificate in certificate chain for CN=<our internal CA>]

Indeed when trying with openssl s_client -connect foreman.example.com:8140, the same error is displayed.

CONNECTED(00000003)
depth=2 CN = Puppet Root CA: 7f5d08af7c925e
verify error:num=19:self signed certificate in certificate chain
verify return:1

If I add the puppet CA to this command itā€™s ok : openssl s_client -CAfile /etc/puppetlabs/puppet/ssl/certs/ca.pem -connect foreman.example.com:8140

SSL handshake has read 5601 bytes and written 652 bytes
Verification: OK

But with the CA trust store, error again :

openssl s_client -CAfile /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt -connect foreman.example.com.ch:8140

CONNECTED(00000003)
depth=2 CN = Puppet Root CA: 7f5d08af7c925e
verify error:num=19:self signed certificate in certificate chain

I managed to create a link to the puppet CA cert in /etc/pki/ca-trust/source/anchors, and ran update-ca-trust. Then the openssl check is always ok, but I couldnā€™t find the correct settings for the puppet server to make it work.

I tried to point the ssl_trust_store setting to the ca-bundle-trust.crt file, but the issue is still present.

Any Idea how to fix this ? Did I miss an option in the installer ?

Expected outcome:

Puppet server should work with custom certificates

Foreman and Proxy versions:

Foreman 3.5
Katello 4.7

Foreman and Proxy plugin versions:

Distribution and version:

RHEL 8.7

Other relevant data:

2

Which version of Puppet are you using? Is the server by any chance version 7 (maybe even 6) and the agent version 5? If so, it isnā€™t able to deal with the intermediate CA that Puppetserver has started to use by default.

3

The puppet agent version is 7.23.0-1.el8
I think the first problem is the certificate error at the end of the install, maybe the puppet agent issue will be resolved by fixing this first issue.

I just tried to install it again, but with minimal settings, and I still have that error at the end of the install :

[root@ certs]# foreman-installer --scenario katello --foreman-initial-organization ā€˜Testā€™ --foreman-initial-location ā€˜Testā€™
2023-03-23 10:54:18 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-03-23 10:54:21 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-03-23 10:54:21 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-03-23 10:54:24 [NOTICE] [configure] Starting system configuration.
2023-03-23 10:58:19 [NOTICE] [configure] 250 configuration steps out of 1381 steps complete.
2023-03-23 10:59:41 [NOTICE] [configure] 500 configuration steps out of 1383 steps complete.
2023-03-23 11:02:09 [NOTICE] [configure] 750 configuration steps out of 1388 steps complete.
2023-03-23 11:02:25 [NOTICE] [configure] 1000 configuration steps out of 1411 steps complete.
2023-03-23 11:05:06 [NOTICE] [configure] 1250 configuration steps out of 1411 steps complete.
2023-03-23 11:07:50 [ERROR ] [configure] Error making POST request to Foreman at https://server.fqdn/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://server.fqdn:9090/v2/features Please check the proxy is configured and running on the host.
2023-03-23 11:07:50 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[server fqdn]/ensure: change from ā€˜absentā€™ to ā€˜presentā€™ failed: Error making POST request to Foreman at https://server.fqdn/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://server.fqdn:9090/v2/features Please check the proxy is configured and running on the host.
2023-03-23 11:07:54 [NOTICE] [configure] System configuration has finished.

There were errors detected during install.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/katello.log

4

This is never expected to work: Puppet uses its own CA.

Can you share how youā€™ve configured your client? Like whatā€™s in /etc/puppetlabs/puppet/puppet.conf.

This is a problem, but Iā€™d assume unrelated to the Puppet one. It may signal that your CA bundle is not correct.

5

Sorry but I reverted the VM to a previous clone that I created before running the installer.
Now I installed Foreman 3.5 Katello 4.7 without a puppet server to try to fix the issue at the end of the install, so the puppet.conf is empty.

The error I pasted from puppet was from the Foreman puppet server, client of himself. Iā€™m still in the installation process, so no client yet registered.

What do you mean by ā€œCA bundle is not correctā€ ? How can I check that ?

6

Note that this foreman server weā€™re trying to install is a client of a satellite server.
Donā€™t know if this can lead to configuration errors, but it may be worth mention it.

Most of our servers are RHEL, managed with the Satellite and configured with the integrated puppet server, but we also have a few dozens of ubuntu installed laptops that we also need to manage. The idea was to install a Foreman, with a puppet server. Our puppet code will be synced to this foreman, and a specific puppet module has been created to manage the ubuntu servers, while the foreman is still configured with our RHEL puppet code, and managed with the Satellite.

Another question : A default foreman installation is supposed to contain an integrated smart proxy, right ? Is this smart proxy supposed to be seen in the GUI ? After the installation we cannot see it.

7

Ok, found the reason for the certificate error on foreman integrated proxy, the splunk forwarder agent was running on port 9090ā€¦ should have noticed that there were splunk information on the certificate check with the openssl command.

Installation is fine, integrated proxy up and running and I can see it in the GUI.

My mistake, thanks to people who tried to help

1 Like