Unable to enroll v5.5 puppet clients on foreman v1.24

mooky31 · December 12, 2019, 2:04pm

Problem:
Some OS, OpenBSD in my case, doesn’t have v6 puppet clients.
Newest avaliable is 5.5.
Those clients won’t enroll in a puppet6 server.
According to puppet site, it is supported though

It seems that it has been reported a while ago: Newly provisioned machines not doing Puppet runs

Expected outcome:
Successful enrollment of client

Foreman and Proxy versions:
Foreman 1.24 freshly installed

Foreman and Proxy plugin versions:
None

Distribution and version:
Client : OpenBSD 6.6
Server : Debian 10

Other relevant data:
Client log:
neosmfs# puppet agent -t
Info: Caching certificate for neosmfs.lab-puppet
Error: Could not request certificate: request https://foreman-pierre.lab-puppet:8140//puppet-ca/v1/certificate_revocation_list/ca failed: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Exiting; failed to retrieve certificate and waitforcert is disabled

ekohl · December 12, 2019, 2:24pm

You’re probably running into another issue where we instruct Puppet to generate an intermediate CA (by calling puppetserver ca setup) but Puppet 5 clients can’t handle this. Right now I can’t find another thread but there’s been some more. https://puppet.com/docs/puppetserver/6.0/intermediate_ca.html the last note

mooky31 · December 12, 2019, 2:54pm

I first run this:

    neosmfs# puppet agent -t
    Info: Creating a new SSL key for neosmfs.lab-puppet
    Info: Caching certificate for ca
    Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
    Info: Creating a new SSL certificate request for neosmfs.lab-puppet
    Info: Certificate Request fingerprint (SHA256): 65:C3:D3:4F:BD:0A:3C:C2:50:21:CF:A2:2F:2A:22:BF:72:37:AE:71:EA:2F:B1:3D:7D:22:E4:D3:3C:09:AB:D8
    Info: Caching certificate for ca
    Exiting; no certificate found and waitforcert is disabled

Then sign the sertificate on the web interface, then re-run puppet agent again:

Info: Caching certificate for neosmfs.lab-puppet
Error: Could not request certificate: request https://foreman-pierre.lab-puppet:8140//puppet-ca/v1/certificate_revocation_list/ca failed: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Exiting; failed to retrieve certificate and waitforcert is disabled

Subsequent runs give this:

neosmfs# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Info: Retrieving pluginfacts
Error: /File[/var/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Error: /File[/var/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Info: Retrieving plugin
Error: /File[/var/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Error: /File[/var/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]

I have never set up an intermediate CA…

Anyway, If I lookup in /etc/puppetlabs/puppet/ssl/ca, I found a ca_crt.pem:

root@foreman-pierre:/etc/puppetlabs/puppet/ssl/ca# openssl x509 -in ca_crt.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Puppet Root CA: 55c855dce7d755
        Validity
            Not Before: Dec 11 10:26:29 2019 GMT
            Not After : Dec  8 10:26:31 2034 GMT
        Subject: CN = Puppet CA: foreman-pierre.lab-puppet

But I can’t find this “Puppet Root CA” certificate. I finaly found it at the bottom of the previous file, so I copied it in a separate file to analyze it;

root@foreman-pierre:/etc/puppetlabs/puppet/ssl/ca# openssl x509 -in /tmp/root.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Puppet Root CA: 55c855dce7d755
        Validity
            Not Before: Dec 11 10:26:29 2019 GMT
            Not After : Dec  8 10:26:31 2034 GMT
        Subject: CN = Puppet Root CA: 55c855dce7d755

So it seems you’re right and there is an intermediate CA.
Using your link and some research (as the configuration guide link in your doc is dead), I finally found the instructions here: https://puppet.com/docs/puppetserver/5.3/intermediate_ca_configuration.html#configure-puppet-agent-certificaterevocation-checking

So I ran on the client:

puppet config set --section main certificate_revocation leaf

This put the last line in this file ;

neosmfs# cat /etc/puppetlabs/puppet/puppet.conf
# This file can be used to override the default puppet settings.
# See the following links for more details on what settings are available:
# - https://puppet.com/docs/puppet/latest/config_important_settings.html
# - https://puppet.com/docs/puppet/latest/config_about_settings.html
# - https://puppet.com/docs/puppet/latest/config_file_main.html
# - https://puppet.com/docs/puppet/latest/configuration.html
[main]
server=foreman-pierre.lab-puppet
certificate_revocation = leaf

Result is the same:

neosmfs# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Info: Retrieving pluginfacts
Error: /File[/var/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Error: /File[/var/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Info: Retrieving plugin
Error: /File[/var/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Error: /File[/var/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet Root CA: 55c855dce7d755]

ekohl · December 12, 2019, 3:13pm

The alternative is to wipe the CA certificates and restart puppetserver. It’ll generate a new one without the intermediate. It does mean you need to build new certs for all machines.

mooky31 · December 12, 2019, 4:34pm

I did this :

root@foreman-pierre:/etc/puppetlabs/puppet/ssl# rm -r ca/*
root@foreman-pierre:/etc/puppetlabs/puppet/ssl# systemctl restart puppetserver

Then I rebooted the server to be sure.
It screwed foreman : when I go to Smart proxies - Puppet CA, it says :

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://foreman-pierre.lab-puppet:8443/puppet/ca)

ekohl · December 12, 2019, 4:39pm

You may want to rerun the installer.

mooky31 · December 12, 2019, 5:28pm

I tried :

root@foreman-pierre:~# rm -r /etc/puppetlabs/puppet/ssl/ca/*
root@foreman-pierre:~# systemctl stop puppetserver
root@foreman-pierre:~# systemctl start puppetserver
root@foreman-pierre:~# foreman-installer
Preparing installation Done
  Success!
[...]

Same problem on puppet CA

mooky31 · December 12, 2019, 6:11pm

My humble opinion : the error message I see on the client seems to say that the problem is the auto-signed certificate. The only autosigned certificate is the one of the root CA, not the intermediate CA.
So I don’t think this comes from that cause.

As this is a custom CA, it is by design autosigned. I think the problem is that the SSL client fetching the file is not told to trust the puppet CA.
It does the same when I try to curl the URL:

neosmfs# curl https://foreman-pierre.lab-puppet:8140//puppet-ca/v1/certificate_revocation_list/ca
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Also, the URL is false (notice the double / after the port number) :

neosmfs# curl --insecure  https://foreman-pierre.lab-puppet:8140//puppet-ca/v1/certificate_revocation_list/ca
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing //puppet-ca/v1/certificate_revocation_list/ca. Reason:
<pre>    Not Found</pre></p>
</body>
</html>
neosmfs# curl --insecure  https://foreman-pierre.lab-puppet:8140/puppet-ca/v1/certificate_revocation_list/ca
-----BEGIN X509 CRL-----
MIICqTCBkgIBATANBgkqhkiG9w0BAQsFADAvMS0wKwYDVQQDDCRQdXBwZXQgQ0E6
IGZvcmVtYW4tcGllcnJlLmxhYi1wdXBwZXQXDTE5MTIxMTEwMjYyOVoXDTM0MTIw
ODEwMjYzMVqgLzAtMB8GA1UdIwQYMBaAFHcuE8wAXRgY79wrugtSDA4EGL3qMAoG
A1UdFAQDAgEAMA0GCSqGSIb3DQEBCwUAA4ICAQCcB4/cd2pe9JjTHiNUpsJ7aOvC
wpEdmVS+QYr0R5UaJuhb60JBNcKqwgMXspuDdkHUTweX9T725aqp3DZpBaXfkTG/
2PaJu2yqADXThwEjMWhOYM5bA6Hhaas9k2NpnNkTW6Drgo3Ww7PHCgVZ5zL/BMpR
u259G7I+o4JtE1CoHSUaijOtfUR4zqU/IPSSncCsHljzffMivVpINtUMKECmaWhr
VvWBwKmUhEcD8HuMqi2ZGNYbMAk47eZREl1Oq7zXQgv4jhe9p5MeWolGLpe8nZh3
5ksxkLYVPoETtsqXuth7IIL9qJAMPWdGD4qtW8+MOJatvxSH7SQpguDtUViDxc/i
G8kqE+RjfaSRD9MuLYEZIWk4Dh0cipj4DoNllWNJU0hOQQY5Ahd83qbL8a7nhFUJ
vntTWhO2hWc0EGzPv/RDQTFJTGGUfJPUdZsjarVKEPH14H0UHCMb3AVNwIRAlv+j
fCOa9Gi3vsdOB1gVUSyHTQ59Yb6cvFl2mp9b43Px2+bHoiPnE/uBcGfOk0Ml5KlA
5i+2MO8qqC6BjM11AdZzxseX8q25cOxYCbo+l8SOxZQ5ri3zjWo6YGx90siJydMm
4Z5B5xWmQYU/grOvf0o2yJh5kdJnfan8Ks+TF67MrgjJS8SEt+lPHYO5RGmDSXB3
VAoYhEXZ9PqsV8JwtQ==
-----END X509 CRL-----
-----BEGIN X509 CRL-----
MIICozCBjAIBATANBgkqhkiG9w0BAQsFADApMScwJQYDVQQDDB5QdXBwZXQgUm9v
dCBDQTogNTVjODU1ZGNlN2Q3NTUXDTE5MTIxMTEwMjYyOVoXDTM0MTIwODEwMjYz
MVqgLzAtMB8GA1UdIwQYMBaAFK9pUoc8PETrrSbcaDlOkSTsxFCHMAoGA1UdFAQD
AgEAMA0GCSqGSIb3DQEBCwUAA4ICAQBJLlF1Aogul0r/knRSGObxnEJav9SNcN/4
Fukc+xR52Bqw6AivZTb9yAjoeUgfhoHgfuqcaDWVAnLt7GnO7hIf4fM7j4Hvjfrj
S+rMNhEQlJFHJQ1AOlklhAlhdwDbUlv4vJvnO1IeWoOdpeXJR/zdOE8BrPEzozAP
ZyOcpR1Fd4kWJfAIvUaoXDr0CEYWDTCMWvR8TnSu/yLZ4HEosrFMyu803wZ9N7sz
J0nP1UvUTldANHvxVpgT6iyy+QkwbAUAphR0T3cKuNbvoNcKQVgy6lPYhwV/bYSW
hwoINjQkkBIeBIqYxsi+ogaByhcOQ8z0xiARs1dP/9Km4yjlhCKiT1EL4e+HFHa4
rHDGiDinXhiy1jx5k71I31nwlamH2gVP1QuYBM8Ew/MUXpSKq/c9oaK7wZv6jtjI
7XAFzQzVut4VjIPnqLIQnRKFy9vYHVuKSlx+6K6D619OEhpAtww1kQWmeQvPU2Dd
Xae2sniW1a/f2/+/QGsy0mlkXcub+pWAW1fhUJmePvfRjV7uZAyO92FK8Fdf3q8s
aSdDvMteQgjJr38wFe1sTc1Id2rZRYveF7ZEjACugrfbGvPfukkWcl6ho+jU9VxB
yiqCr50QaPga83wuXGJVgLlHgW9HITKUBMVf68BxJ0Ha5rsnavzIs/2l8DSubPCQ
RLabuaU/OA==
-----END X509 CRL-----

mooky31 · December 12, 2019, 6:30pm

I got some progress copying /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem from the server to /etc/puppetlabs/puppet/ssl/certs/ca.pem on the client. This file only ontained the certificate of the sub CA. Adding the root one lead me to this:

neosmfs# puppet agent -t
Info: Caching certificate_revocation_list for ca
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Notice: /File[/var/puppetlabs/puppet/cache/locales]/group: group changed '_puppet' to 'wheel'
Info: Caching catalog for neosmfs.lab-puppet
Info: Applying configuration version '1576174363'
Info: Creating state file /var/puppetlabs/puppet/cache/state/state.yaml
Notice: Applied catalog in 0.01 seconds

The remaining warning must come from the wrong URL for the CRL.

mooky31 · December 13, 2019, 8:10am

I don’t know why, but when I retried this morning:

neosmfs# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Info: Retrieving pluginfacts
Error: /File[/var/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Error: /File[/var/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Info: Retrieving plugin
Error: /File[/var/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Error: /File[/var/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]

I reverted to snapshot, repeated the same steps, and was unable to make it work again.
After modifying the CA manually it fails on the CRL because of the bad URL:

neosmfs# puppet agent -t
Info: Caching certificate for neosmfs.lab-puppet
Info: Caching certificate_revocation_list for ca
Error: Could not request certificate: request https://foreman-pierre.lab-puppet:8140//puppet-ca/v1/certificate/neosmfs.lab-puppet failed: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get certificate CRL): [unable to get certificate CRL for /CN=Puppet CA: foreman-pierre.lab-puppet]
Exiting; failed to retrieve certificate and waitforcert is disabled

I am starting to desperate, this is the very first step of using formean/puppet and it fails…

Beam · April 1, 2020, 9:17am

Any fixes, progress?

I ran into the same problem.

mooky31 · April 1, 2020, 10:04am

I ended using Ansible, no more problems with the agent

shanlej · April 16, 2020, 2:48pm

@Beam - I have also encountered this problem.
I am creating a new Puppet 6 environment and wish to migrate agents running Puppet 5.

I followed the instructions in this thread as @mooky31 has correctly identified the key steps:

1 - Replace the /etc/puppetlabs/puppet/ssl/ca.pem on the Puppet v5 agent with the contents of: /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem which resides on the CA server. The ca_crt.pem contains both the root CA and the intermediate CA that were generated as part of Puppet 6 CA setup.
This will get rid of the certificate verify error and now leave an “unable to get certificate CRL” error. To fix this…
2 - Execute the following command on the Puppet v5 agent: puppet config set --section main certificate_revocation leaf.

In our scenario we will need to push these changes out using a Puppet module to several thousand Puppet v5 agents.

@mooky31 - great thread - very helpful.