Newly provisioned machines not doing Puppet runs

wings · June 27, 2019, 2:35am

Problem:
On my fresh 1.22.0 installation, newly provisioned machines successfully install but never generate a report and don’t appear to be doing Puppet runs successfully.

When running the agent manually on those machines, we see this error:

[root@demo7 ~]# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://fm-master3.riff.cc/pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://fm-master3.riff.cc/plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: a3558aa6a0a245]

Expected outcome:
Puppet should run without issue.

Foreman and Proxy versions:
1.22.0

Foreman and Proxy plugin versions:
1.22.0

wings · June 27, 2019, 6:26am

This is on CentOS 7, fully patched, firewall off (for testing only).

Dirk · June 27, 2019, 6:36am

Can you tell me which Puppetserver version is installed on the master and Puppet version on the systems?

I think I saw similar errors when Puppet 3.7 from CentOS repositories was still used with Puppetserver 5 from Puppet repositories on the master. If this is the case you can install Puppet 5 during provisioning by setting the parameter enable-puppetlabs-puppet5-repo to true.

wings · June 27, 2019, 6:48am

On the master:

[wings@fm-master3 ~]$ puppetserver --version
puppetserver version: 6.4.0

On the client:

[root@demo7 ~]# puppet --version
3.6.2

Very weird. Is there a way to enable Puppet6 provisioning on these machines?

Dirk · June 27, 2019, 7:05am

At the moment no, but he templates could be easily extended similar to Puppet 5:

github.com

theforeman/community-templates/blob/develop/provisioning_templates/provision/kickstart_default.erb

<%#
kind: provision
name: Kickstart default
model: ProvisioningTemplate
oses:
- CentOS
- Fedora
- RedHat
%>
<%#
This template accepts the following parameters:
- lang: string (default="en_US.UTF-8")
- selinux-mode: string (default="enforcing")
- keyboard: string (default="us")
- time-zone: string (default="UTC")
- http-proxy: string (default="")
- http-proxy-port: string (default="")
- force-puppet: boolean (default=false)
- enable-epel: boolean (default=true)
- enable-puppetlabs-repo: boolean (default=false)

This file has been truncated. show original

github.com

theforeman/community-templates/blob/develop/provisioning_templates/snippet/puppetlabs_repo.erb

<%#
kind: snippet
name: puppetlabs_repo
model: ProvisioningTemplate
snippet: true
-%>
<%
http_proxy   = host_param('http-proxy') ? " --httpproxy #{host_param('http-proxy')}" : nil
http_port    = host_param('http-proxy-port') ? " --httpport #{host_param('http-proxy-port')}" : nil
proxy_uri    = host_param('http-proxy') ? "http://#{host_param('http-proxy')}:#{host_param('http-proxy-port')}" : nil
proxy_string = proxy_uri ? " -e https_proxy=#{proxy_uri}/" : ''
proxy_string_bits = proxy_uri ? " -ProxyUsage Override -ProxyList #{proxy_uri}" : ''
os_family = @host.operatingsystem.family
os_major  = @host.operatingsystem.major.to_i
os_name   = @host.operatingsystem.name

if os_family == 'Redhat'
  repo_host = 'yum.puppetlabs.com'
  if os_name == 'Fedora'
    repo_os = 'fedora'

This file has been truncated. show original

github.com

theforeman/community-templates/blob/develop/provisioning_templates/snippet/puppet_setup.erb

<%#
kind: snippet
name: puppet_setup
model: ProvisioningTemplate
description: this snippet will configure the Puppet agent
snippet: true
%>
<%
os_family = @host.operatingsystem.family
os_major  = @host.operatingsystem.major.to_i
os_name   = @host.operatingsystem.name

aio_enabled = host_param_true?('enable-puppetlabs-puppet5-repo') || host_param_true?('enable-puppetlabs-pc1-repo') || host_param_true?('enable-puppet4') || host_param_true?('enable-puppet5')

if os_family == 'Freebsd'
  freebsd_package = host_param_true?('enable-puppet5') ? 'puppet5' : 'puppet4'
  etc_path = '/usr/local/etc/puppet'
  bin_path = '/usr/local/bin'
elsif os_family == 'Windows'
  windows_package = "puppet-agent-#{@host.architecture}.msi"

This file has been truncated. show original

wings · June 27, 2019, 7:18am

I gathered some more info. My Ubuntu client to that server (which gives the same errors as in my first post):

root@demo8:~# puppet --version
5.4.0

Then my old master running 1.20.2:

wings@fm-master2:~$ puppetserver --version
puppetserver version: 5.3.8

And its CentOS client:

[root@demo1 ~]# puppet --version
3.6.2

And its Ubuntu client:

root@demo2:~# puppet --version
5.4.0

On my 1.20.2 master, things work without issue including reporting.

wings · July 9, 2019, 12:24am

Still having issues… 1.20.2 works perfectly, unfortunately this is our last major blocker before the next stage…

wings · July 15, 2019, 6:22am

I went ahead and deployed a fresh 1.22.0 master, but installed Puppet 5 instead of Puppet 6 at the very start. Doing this, it worked perfectly (nodes successfully checking in and completing runs) so it appears there’s been a regression with the Puppet 6 master.

I’ve got both working and ready for testing, so if there’s anything I can do to help get this fixed let me know. This does buy some time though, as we were likely to stick with Puppet 5 for at least a couple more months.

ekohl · July 15, 2019, 9:40am

I had a stab at this, but haven’t tested it yet. Would be great if anyone could have a look.
https://github.com/theforeman/community-templates/pull/597

Stanislav_Kopp · July 31, 2019, 3:47pm

I have the same issue with puppetserver 6 and foreman 1.22. We still have some old Debian Wheezy where only puppet agent 5 is supported. Are there any workarounds?

wings · August 6, 2019, 5:07am

This is what I did to downgrade to Puppet5 master. YMMV and please be careful. I’d suggest doing backups before following this procedure. You may not need to do katello stuff if you’re not using katello -

Remove all Puppet packages
Remove Puppet6 repo
Add Puppet5 repo
Install katello and foreman-installer packages.
Re-run our scenario:

sudo foreman-installer --scenario katello \
  --enable-foreman-proxy \
  --foreman-proxy-tftp=true \
  --foreman-proxy-tftp-servername=10.20.30.95 \
  --foreman-proxy-dhcp=true \
  --foreman-proxy-dhcp-interface=eth0 \
  --foreman-proxy-dhcp-gateway="10.20.30.1" \
  --foreman-proxy-dhcp-nameservers="10.20.30.6,10.20.30.7"

Run the resetting parameters

foreman-installer --reset-puppet-server-jvm-extra-args --reset-puppet-server-puppetserver-version --reset-puppet-server-puppetserver-metrics --reset-puppet-server-puppetserver-experimental

From the rest of my notes:

remove all Puppet packages + puppet-dependent packages including katello + foreman-installer (which will be removed when you do this whether you like it or not)
• remove Puppet6 repo package
• remove /etc/puppetlabs folder (I datestamped it and moved it to /root/) (probably overkill)
• install Puppet5 repo package
• install katello + foreman-installer and let them pull in puppet5 puppetserver
• re-run the scenario command (might not be necessary)
• reset puppet stuff in foreman
foreman-installer --reset-puppet-server-jvm-extra-args --reset-puppet-server-puppetserver-version --reset-puppet-server-puppetserver-metrics --reset-puppet-server-puppetserver-experimental (edited)

mooky31 · December 12, 2019, 1:54pm

Hello,

The problem is still present on a freshly installed v1.24.
Is there any fix ? Downgrading doesn’t seem a reliable solution…