After updating Foreman, I have had SSL issues when machines contact puppet to pull configuration.
2021-06-15T11:25:13.847+01:00 WARN [qtp1169515155-36] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During fact upload occured an exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed
Serving cached ENC: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed
This problem seems fairly similar to the following issue:
I have had to disable #:ssl_ca: in /etc/puppetlabs/puppet/foreman.yaml to allow hosts to bypass the ca check. Each time I update Foreman this problem returns and I have to disable the ssl_ca again.
I run /etc/puppetlabs/puppet/node.rb test-node.com which returns all the facts associated with the host. Given some of the other comments on the link above, this seems very much a problem with Foreman having issues with the Puppet certs.
One of the bits of advice here was to regenerate all the puppet certs which seemed to be the nuclear option. This could be a pretty messy too, especially if it was just the configuration that was wrong.
That’s probably because the installer always resets the file. I assume you use different CA for Foreman (with Katello?) and your puppet infrastructure. It should still be possible to configure certificates so it works. You need to make sure that /etc/puppetlabs/puppet/foreman.yaml has the cert of the CA that signed Foreman’s certificate. If you use katello, it’s typically located at /etc/pki/katello/certs/katello-server-ca.crt. You can find out what certificates the Foreman server uses in this file 05-foreman-ssl.conf.
Note that there’s a second part of the equation, Foreman must also trust the client certificate that the node.rb uses. That means Foreman must trust the CA that created that client cert, in this case most likely the Puppet CA.
If you find the right combination of certificates to use, make sure to also configure it via the installer, otherwise every update of Foreman will reset the config files.
When I initially setup this foreman instance (not katello), I don’t remember doing anything too adventurous. Aside from configuring a signed SSL/TLS certs for the main address. Not sure if I did something wrong at this stage and that is why I’m running into issues here.
My understanding was this CA cert was used for foreman proxy to communicate with puppet server and should match. From what you have mentioned around the 05-foreman-ssl.conf file, I’m not sure if this understanding is correct.
I had checked the foreman.yaml file you mentioned and the certs it points to, and they didn’t seem to change between updates. I had also tried configuring other CA certs a while ago, but had no luck there.
Foreman does not use the system store if ssl_ca file is specified. That’s how openssl library work, if explicit file is specified, it only trust that.
Looking at your configs, it makes sense to me. You deployed a custome certificate for the Foreman web app. I assume the issuer of /etc/pki/tls/certs/lnx-foreman.somewhere.com/lnx-foreman.somewhere.com.crt is not your Puppet CA. Therefore the node.rb that only trusts certifficates issued by /etc/pki/tls/certs/lnx-foreman.somewhere.com/lnx-foreman.somewhere.com.crt fails to verify the Foreman app authenticity. The node.rb script talks to Foreman directly, though it’s deployed on the same box as Foreman Proxy and in most cases also the same box as Foreman itself.
If all my assumptions are correct, you should get the cert of the issuer of /etc/pki/tls/certs/lnx-foreman.somewhere.com/lnx-foreman.somewhere.com.crt and append it to /etc/puppetlabs/puppet/ssl/certs/ca.pem or even better, point foreman.yaml :ssl_ca: to that certificate explicitly.
Hi Merek, the red fedora in your picture has obviously been won in battle, you have solved my issue.
I had played around assigning different certs to :ssl_ca:, I hadn’t tried chaining them together. Once I had joined foreman CA with the intermediary and root certs into a chain, config started to push out correctly.
In terms of the cause, I wondering if this something to do with the way I have been updating Foreman. When I run foreman-installer during the update steps, should I continue to with options that include my custom cert?
Glad it helped! I’m not sure what is the magic combination of the installer arguments to use custom certificate for the web app, perhaps @ehelms would know. I think some improvements in this area are being worked on, so hopefully this will get easier at some point.
That’s fine. The next time I update I will try a few different arguments. I’ll snapshot the VM so I can replay the process a few times to see what happens.
If you have the same issue as I did, your /etc/puppetlabs/puppet/ssl/certs/ca.pem doesn’t contain the correct certificates to allow the ssl_cert to validate a secure connection.
Check on your SSL provider’s website/emails for intermediate/root certs, these need to be configured with your foreman instance.
Backup /etc/puppetlabs/puppet/ssl/certs/ca.pem to somewhere, replace the contents of this file with the intermediate and root certificates. Run the puppet agent on one of your machines, this should hopefully fix the problem for you.
