Newly provisioned machines not doing Puppet runs

Problem:
On my fresh 1.22.0 installation, newly provisioned machines successfully install but never generate a report and don’t appear to be doing Puppet runs successfully.

When running the agent manually on those machines, we see this error:

[root@demo7 ~]# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://fm-master3.riff.cc/pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://fm-master3.riff.cc/plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]

Expected outcome:
Puppet should run without issue.

Foreman and Proxy versions:
1.22.0

Foreman and Proxy plugin versions:
1.22.0

This is on CentOS 7, fully patched, firewall off (for testing only).

Can you tell me which Puppetserver version is installed on the master and Puppet version on the systems?

I think I saw similar errors when Puppet 3.7 from CentOS repositories was still used with Puppetserver 5 from Puppet repositories on the master. If this is the case you can install Puppet 5 during provisioning by setting the parameter enable-puppetlabs-puppet5-repo to true.

On the master:

[wings@fm-master3 ~]$ puppetserver --version
puppetserver version: 6.4.0

On the client:

[root@demo7 ~]# puppet --version
3.6.2

Very weird. Is there a way to enable Puppet6 provisioning on these machines?

At the moment no, but he templates could be easily extended similar to Puppet 5:



I gathered some more info. My Ubuntu client to that server (which gives the same errors as in my first post):

root@demo8:~# puppet --version
5.4.0

Then my old master running 1.20.2:

wings@fm-master2:~$ puppetserver --version
puppetserver version: 5.3.8

And its CentOS client:

[root@demo1 ~]# puppet --version
3.6.2

And its Ubuntu client:

root@demo2:~# puppet --version
5.4.0

On my 1.20.2 master, things work without issue including reporting.

Still having issues… 1.20.2 works perfectly, unfortunately this is our last major blocker before the next stage…

I went ahead and deployed a fresh 1.22.0 master, but installed Puppet 5 instead of Puppet 6 at the very start. Doing this, it worked perfectly (nodes successfully checking in and completing runs) so it appears there’s been a regression with the Puppet 6 master.

I’ve got both working and ready for testing, so if there’s anything I can do to help get this fixed let me know. This does buy some time though, as we were likely to stick with Puppet 5 for at least a couple more months.

I had a stab at this, but haven’t tested it yet. Would be great if anyone could have a look.

1 Like

I have the same issue with puppetserver 6 and foreman 1.22. We still have some old Debian Wheezy where only puppet agent 5 is supported. Are there any workarounds?

This is what I did to downgrade to Puppet5 master. YMMV and please be careful. I’d suggest doing backups before following this procedure. You may not need to do katello stuff if you’re not using katello -

  • Remove all Puppet packages
  • Remove Puppet6 repo
  • Add Puppet5 repo
  • Install katello and foreman-installer packages.
  • Re-run our scenario:
sudo foreman-installer --scenario katello \
  --enable-foreman-proxy \
  --foreman-proxy-tftp=true \
  --foreman-proxy-tftp-servername=10.20.30.95 \
  --foreman-proxy-dhcp=true \
  --foreman-proxy-dhcp-interface=eth0 \
  --foreman-proxy-dhcp-gateway="10.20.30.1" \
  --foreman-proxy-dhcp-nameservers="10.20.30.6,10.20.30.7"
  • Run the resetting parameters
foreman-installer --reset-puppet-server-jvm-extra-args --reset-puppet-server-puppetserver-version --reset-puppet-server-puppetserver-metrics --reset-puppet-server-puppetserver-experimental

From the rest of my notes:

  • remove all Puppet packages + puppet-dependent packages including katello + foreman-installer (which will be removed when you do this whether you like it or not)
    • remove Puppet6 repo package
    • remove /etc/puppetlabs folder (I datestamped it and moved it to /root/) (probably overkill)
    • install Puppet5 repo package
    • install katello + foreman-installer and let them pull in puppet5 puppetserver
    • re-run the scenario command (might not be necessary)
    • reset puppet stuff in foreman
    foreman-installer --reset-puppet-server-jvm-extra-args --reset-puppet-server-puppetserver-version --reset-puppet-server-puppetserver-metrics --reset-puppet-server-puppetserver-experimental (edited)