Foreman 1.19, Puppet (5.5.6), Centos 7: "Certificate has expired for /CN=Puppet CA"

I bid you good fortune.

On inherited Foreman 1.19 / Puppet (5.5.6) presence, the original, 5-year cert (CA) expired. I found out while trying to run puppet agent on a new node.
There was no complete documented series of steps for this combo of versions.

I followed a combination of what I saw:

On puppet master:

rm -rf /etc/puppetlabs/puppetdb/ssl
puppetserver ca generate --config /etc/puppetlabs/puppet/puppet.conf
puppet master --no-daemonize --verbose
puppet resource service puppetserver ensure=running
puppet resource service puppet ensure=running

/opt/puppetlabs/server/apps/puppetdb/bin/puppetdb ssl-setup -f
puppet resource service puppetdb ensure=running

But on the master itself, and certainly a separate client/agent, there are any number or errors.

  • If on puppet master, running “puppet agent -t”, I get this this error, where should I be checking?

Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using ‘eval_generate’: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: *******]
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: ***]

  • On puppet master, “systemctl status -l puppet” concludes in
    “Unable to fetch my node definition, but the agent run will continue:”
    “Sep 28 16:58:42 puppet-agent[19623]: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: ********]”
    “Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: ******]”

  • If I run the same on any client, with below result, where should I head?

Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get issuer certificate for /CN=Puppet CA: ]
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using ‘eval_generate’: SSL_connect returned=1 errno=0 s
tate=error: certificate verify failed: [unable to get issuer certificate for /CN=Puppet CA: ]
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get issuer certificate for /CN=Puppet CA: ]
There is an existing “foreman-answers.yaml” in use all these years. I found out recently that Foreman created all the puppet items. But I don’t know that “foreman-install” preserves it, in favor of fixing anything here.

I know the software versions are rather old; upgrading not an option until this impasse can be resolved.

Any leads or suggested next steps would be appreciated.

I re-traced using this reference: https://alexshepherd.me/posts/changing-foremans-ssl-certificate/

  • “server_ssl_cert” (/etc/foreman-installer/scenarios.d/foreman-answers.yaml) does already match SSLCertificateFile (/etc/httpd/conf.d/05-foreman-ssl.conf)

  • “server_ssl_chain” from above is one of the files generated from the recent puppet action (puppetserver ca generate --config) It does match “SSLCertificateChainFile” on the latter SSL conf.

  • “server_ssl_key” (/etc/foreman-installer/scenarios.d/foreman-answers.yaml) already matches SSLCertificateKeyFile (/etc/httpd/conf.d/05-foreman-ssl.conf)

  • “puppet_ssl_ca” /etc/puppetlabs/puppet/ssl/certs/ca.pem; that matches “server_ssl_chain” /etc/puppetlabs/puppet/ssl/certs/ca.pem
    But it does NOT match “server_ssl_chain_filepath” (/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem) Does that matter? It is present in the foreman-answers.yaml.

the first thing I’d do before fixing this (it’s already broken)

is either a.) move puppet and foreman to supported releases
b.) build a new foreman puppet server (or appropriate replacement tech) and plan a client migration

I don’t have a 1.9 host available to me to validate the menu options, but in the infrastructure tab → smart proxies there should be a tab for puppetca - what does that think is happening as that is what the clients interact with / foreman interacts with the client, so fixing anything behind that is important, but if foreman doesn’t know what’s going on, you’re still going to broken.

Thank you for reply!
At "Smart Proxies > “Puppet CA” are these pink-highlighted errors:

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for proxy https://hydra.igm.cumc.columbia.edu:8443/puppet/ca)

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for proxy https://hydra.igm.cumc.columbia.edu:8443/puppet/ca)

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-3517 [ProxyAPI::ProxyException]: Unable to get PuppetCA autosign ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for proxy https://hydra.igm.cumc.columbia.edu:8443/puppet/ca)

so you’re using the puppet cert and CA to talk between foreman and the smart proxies too.that’s why your puppet run is initially failing as it’s talking through the smart proxy

Again - I’d look at upgrading this too puppet 7 and foreman 3, then resolving this, the which will allow you to have a longer opportunity to keep using this host while you work out your medium/long term plans for it.

I had notes on the foreman → smart proxy certificate chain as I never used to use self signed certs from the puppet CA, and at the time the 1.x installer didn’t really have the installer options maturity for this to work out of the box, I will try to find these notes, but the first task is to get the foreman instance and smart proxy talking to each other cleanly, as without that very little else matters if it’s working or not.

I’ve not got a 1.x instance to hand or puppet 5 to check all the file paths so I suspect any information you can provide when trying to fix this on what you’re changing in what file paths and why would help people who don’t have the same legacy install layout to hand to work with you