Renew CA cert for puppet / foreman

foremanrox · March 28, 2023, 5:25pm

Hey so i think i got a pretty major problem, the cert for puppet seems to have expired last week and now foreman and puppet will no longer run on any workstation.

Using the proceedure here: Renewing CA certificates procedure I attempted to renew the puppetCA cert. It did seem to work, but did not fix my clients. I restarted the foreman server and then i could not log in to the backend of foreman (webpage), so had to revert to snapshot. However the webserver certificate had been updated (new expiry 2033), so i think i was successful regenerating the certs. However like i said, my username and pass no longer worked on the backend and so i had to revert. Not sure why that happened…

Anyone got some more complete instructions? because i assume i am missing something. I basically followed this guide ( Renew/Extend Puppet CA/puppetmasterd certs | Arrfab's blog ) substituting /etc/puppetlabs/puppet/ssl/ with the path they were using

client error is:

If anyone knows a client config that will just bypass certificates in general that would be a solid work around too. I just need this to work. I have a feeling i am pretty stuck as I wont have any way to distribute any change to the clients (or i have to come up with a way like group policy…)

Foreman version is 3.1.2

here is the instructions i was drafting, everything i did so far to try and fix this. Hopefully i am just missing one small step and its not as bad as i think it is

1.) back up old certificates just incase (can do snapshot as well in vmware)

tar cvzf /root/puppet-ssl-backup.2023.03.28.gz /etc/puppetlabs/puppet/ssl/

2.) Check that the ca_key.pem and existing ca_crt.pem match

cd /etc/puppetlabs/puppet/ssl/ca
( openssl rsa -noout -modulus -in ca_key.pem 2> /dev/null | openssl md5 ; openssl x509 -noout -modulus -in ca_crt.pem 2> /dev/null | openssl md5 )

3.) Generate a new CSR.

openssl x509 -x509toreq -in ca_crt.pem -signkey ca_key.pem -out ca_csr.pem

4.) sign the CSR for the CA

cat > extension.cnf << EOF
[CA_extensions]
basicConstraints = critical,CA:TRUE
nsComment = “Puppet Foreman Internal Certificate”
keyUsage = critical,keyCertSign,cRLSign
subjectKeyIdentifier = hash
EOF

5.) And now archive old CA crt and sign (new) extended one.

cp ca_crt.pem ca_crt.pem.old

openssl x509 -req -days 3650 -in ca_csr.pem -signkey ca_key.pem -out ca_crt.pem -extfile extension.cnf -extensions CA_extensions

6.) check the cert

openssl x509 -in ca_crt.pem -noout -text|grep -A 3 Validity

7.) Puppetmasterd server: We have also to regen the CSR from the existing cert (assuming our fqdn for our cert is correctly also the currently set hostname)

cd /etc/puppetlabs/puppet/ssl

openssl x509 -x509toreq -in certs/foremanDOMAIN.pem -signkey private_keys/foremanDOMAIN.pem -out certificate_requests/foremanDOMAIN_csr.pem

8.) Sign it with the new CA

cp certs/foremanDOMAIN.pem certs/foremanDOMAIN.pem.old #Backing up

openssl x509 -req -days 3650 -in certificate_requests/foremanDOMAIN_csr.pem -CA ca/ca_crt.pem
-CAkey ca/ca_key.pem -CAserial ca/serial -out certs/foremanDOMAIN.pem

9.) Validating that puppetmasted key and new certs are matching (so crt and private keys are ok)

( openssl rsa -noout -modulus -in private_keys/foremanDOMAIN.pem 2> /dev/null | openssl md5 ; openssl x509 -noout -modulus -in certs/foremanDOMAIN.pem 2> /dev/null | openssl md5 )

10.) restart puppetmasterd

systemctl restart puppet

** so far did not work ***

any help appreciated. We have no foreman administrator anymore, so its just me trying to figure shit out… I may try the cert update again and then look at the logs to see why i cant log in…

foremanrox · March 28, 2023, 5:54pm

EDIT: i did the same thing again and now i can login with my credentials… still hasnt fixed the root problem but at least i can still log into foreman.

foremanrox · March 28, 2023, 6:47pm

Error displayed on the web GUI when doing a "refresh features"

Danger alert:Error
Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) for proxy https://foreman.DOMAIN:8443/v2/features and Please check the proxy is configured and running on the host.

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) for proxy https://foreman.DOMAIN:8443/puppet/ca)

From foreman /var/log/foreman-proxy/proxy.log

2023-03-28T10:38:25 [I]
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 4355 (0x1103)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Puppet CA: foreman.ms.sd40.bc.ca
Validity
Not Before: Mar 28 17:35:28 2023 GMT
Not After : Mar 25 17:35:28 2033 GMT
Subject: CN=foreman.DOMAIN.ca
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)

…

2023-03-28T10:39:05 [E] OpenSSL::SSL::SSLError SSL_accept returned=1 errno=0 state=error: sslv3 alert certificate expired
/usr/lib/ruby/2.5.0/webrick/server.rb:299:in accept' /usr/lib/ruby/2.5.0/webrick/server.rb:299:in block (2 levels) in start_thread’
/usr/lib/ruby/2.5.0/webrick/utils.rb:263:in timeout' /usr/lib/ruby/2.5.0/webrick/server.rb:297:in block in start_thread’
/usr/lib/ruby/vendor_ruby/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’
2023-03-28T10:39:08 [E] OpenSSL::SSL::SSLError SSL_accept returned=1 errno=0 state=error: sslv3 alert certificate expired

it does seem to load the correct cert, i just dont know why it isnt using it… Seems to be the error below which i forget where i got it from might be signficant:

state=error: tlsv1 alert unknown ca puppet

foremanrox · March 28, 2023, 9:05pm

I think it might have to do with the following file: :puppet_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem

I had assumed this file and the HOSTNAME.pem file were identical, but i looked at the old server files and they are different certs! i do not know how to regenerate the CA.PEM file… that may be the key here. as it is not covered in the guide and still has an old date of 2018.

So the hostname one has the CN being: Puppet CA: foreman.DOMAIN
and the CA.PEM file has the CN being: foreman.DOMAIN and it has a SAN as well: foreman.DOMAIN, puppet, puppet.DOMAIN

ill see if i can figure out how to regenerate this second certificate, ca.pem.

foremanrox · March 28, 2023, 9:55pm

haha! using this guide Renewing Puppet CA and puppet master certificates i have fixed the server errors! my puppet proxy now works somewhat. As in there are no errors on the server anymore. However, my clients still cannot puppet agent -t with the same message…
for some reason it weirdly only pushed the date back by 3 months till july… so probably not fixed, but its a good place to start at tomorrow i think. all my server side errors seem to be gone… but same “cert expired” message on the clients. Well it feels like a victory after spending all day on this!

foremanrox · March 28, 2023, 10:58pm

Ok! last update today as its time to go home. I found out now that if i simply replace “C:\ProgramData\PuppetLabs\puppet\etc\ssl\certs\ca.pem” on the client from “/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem” on the server, the client connects fine! no certificate regenerations needed on the client side!

ill have to test a bit more, but surely i can use gpo to push out one single file maybe this has done it.

of course if anyone else has any better suggestions let me know! thanks for joining me on this journey, and i hope it helps others… We were debating ditching foreman because of this today. Sure its my fault that the certificate expired, but i mean come on. There’s like a million certificates to keep track of in any company, and i never met a sysadmin who actually likes certificates or changes them all out proactively!! Generally most people wait till something breaks It would be nice if foreman had a way to disable certificate checking, incase problems like this occured.

foremanrox · March 29, 2023, 5:57pm

Ok just to close the loop, here is the full set of instructions that i did to get this working.

1.) First, copy the important files to a new directory

mkdir /root/puppet_renewal
cd /root/puppet_renewal
mkdir /root/puppet_renewal/ca
mkdir /root/puppet_renewal/puppetmaster
mkdir /root/puppet_renewal/puppetmaster/private_keys
mkdir /root/puppet_renewal/puppetmaster/certs
cp /etc/puppetlabs/puppet/ssl/ca/ca_key.pem /root/puppet_renewal/ca
cp /etc/puppetlabs/puppet/ssl/private_keys/foreman.DOMAIN.pem /root/puppet_renewal/puppetmaster/private_keys

2.) Get the important information from the existing CA and puppet master certificate. Run the following commands and paste the info below. (these might be different when you run it so take note)

openssl x509 -in /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem -noout -subject

subject=CN = Puppet CA: foreman.DOMAIN

openssl x509 -in /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem -noout -serial

serial=01

( The below returns nothing for me, idk…)

openssl x509 -in /etc/puppetlabs/puppet/ssl/certs/foreman.DOMAIN.pem -noout -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux

    X509v3 extensions:
        X509v3 Basic Constraints:
            CA:FALSE
        X509v3 Subject Key Identifier:
            9B:17:02:F9:36:44:C7:42:F9:67:45:70:55:C2:AD:FB:6F:83:2B:0A
        Netscape Comment:
            Puppet Ruby/OpenSSL Internal Certificate
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Alternative Name:
            DNS:foreman.DOMAIN, DNS:puppet, DNS:puppet.mydomain

3.) Make the openssl config file:

cat << EOF > /root/puppet_renewal/renewpuppet.cnf
[ v3_ca ]
basicConstraints= CA:TRUE
subjectKeyIdentifier= hash
#authorityKeyIdentifier= keyid:always,issuer:always
keyUsage = critical, cRLSign, keyCertSign
nsComment = ‘Puppet Ruby/OpenSSL Internal Certificate’

[ v3 ]
basicConstraints= CA:FALSE
subjectKeyIdentifier= hash
nsComment = ‘Puppet Ruby/OpenSSL Internal Certificate’
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = critical, serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = foreman.DOMAIN
DNS.2 = puppet
DNS.3 = puppet.DOMAIN
EOF

4.) Make a CSR using your existing CA key, and make the new CA cert. take note of the serial number and subject name from step 2.

openssl req -out /root/puppet_renewal/ca/ca_new.csr -key /root/puppet_renewal/ca/ca_key.pem -new -batch -subj "/CN=Puppet CA: foreman.DOMAIN"

openssl x509 -req -days 3650 -in /root/puppet_renewal/ca/ca_new.csr -signkey /root/puppet_renewal/ca/ca_key.pem -out /root/puppet_renewal/ca/ca_crt.pem -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3_ca -set_serial 1

5.) Find the next serial number for the existing CA

echo $((0xcat /etc/puppetlabs/puppet/ssl/ca/serial))

4354

6.) Make a CSR using your existing puppet server key, and make the new puppet server certificate. Use the serial number from step 5 below.

openssl req -out /root/puppet_renewal/mypuppetmaster.csr -key /root/puppet_renewal/puppetmaster/private_keys/foreman.DOMAIN.pem -new -batch -subj “/CN=foreman.DOMAIN”

openssl x509 -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3 -req -days 1825 -in /root/puppet_renewal/mypuppetmaster.csr -CA /root/puppet_renewal/ca/ca_crt.pem -CAkey /root/puppet_renewal/ca/ca_key.pem -CAcreateserial -out /root/puppet_renewal/puppetmaster/certs/foreman.DOMAIN.pem -sha256 -set_serial 4354

7.) So, now you have an updated CA certificate, and the new puppetmaster cert. Copy them into place, and restart the service

systemctl stop apache2

cp /root/puppet_renewal/ca/ca_crt.pem /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem
cp /root/puppet_renewal/ca/ca_crt.pem /etc/puppetlabs/puppet/ssl/certs/ca.pem
cp /root/puppet_renewal/puppetmaster/certs/foreman.DOMAIN.pem /etc/puppetlabs/puppet/ssl/certs/foreman.DOMAIN.pem

systemcrl start apache2

8.) make a GPO that replaces the ca.pem file in the windows directory C:\ProgramData\PuppetLabs\puppet\etc\ssl\certs . This will allow the client to now connect. A file replace (run once) gpo is all it took there.

9.) reboot the entire server just to be sure everything is good. Your puppet CA smart proxy should show all green now and as the clients run the GPO, the clients will start popping back into foreman just like nothing has happened.

ref: Renewing Puppet CA and puppet master certificates