Foreman, Katello, Puppet. Scalable build. Node failing following CSR

Support
#1

G'day everyone.

So far, have built out:

  • *host001 *- PostgreSQL
  • *host002 *- memcached
  • *host003 *- Foreman UI with Katello
  • *host004 *- Puppet certificate authority
  • *host005 *- Puppet compile master

Where:

  • *host003 *uses *host001 *for persistent storage, *host002 *for
    transient values (in the event we create more Foreman nodes)
  • *host004 *is connected to *host003 *using Smart Proxy
  • *host005 *is connected to *host003 *using Smart Proxy

If we take a new host - *node001 *- and modify puppet.conf to read like

  • [main]*

  • ca_server = host004*

  • [agent]*

  • server = host005*

And execute puppet agent --test --verbose, we can successfully begin a
certificate signing request.

After we sign the request on host004, we execute puppet agent --test
–verbose from node001 again, but encounter the following:

[root@node001 ~]# puppet agent --test --verbose
Warning: Unable to fetch my node definition, but the agent run will
continue:

Warning: SSL_connect returned=1 errno=0 state=error: certificate verify
failed: [self signed certificate in certificate chain for /CN=Puppet CA:
host005]

Info: Retrieving pluginfacts

Error: /File[/opt/puppetlabs/puppet/facts.d]: Failed to generate
additional resources using 'eval_generate': SSL_connect returned=1 errno=0
state=error: certificate verify failed: [self signed certificate in
certificate chain for /CN=Puppet CA: host005]

Error: /File[/opt/puppetlabs/puppet/facts.d]: Could not evaluate: Could
not retrieve file metadata for puppet:///pluginfacts: SSL_connect
returned=1 errno=0 state=error: certificate verify failed: [self signed
certificate in certificate chain for /CN=Puppet CA: host005]

Info: Retrieving plugin

Error: /File[/opt/puppetlabs/puppet/lib]: Failed to generate additional
resources using 'eval_generate': SSL_connect returned=1 errno=0
state=error: certificate verify failed: [self signed certificate in
certificate chain for /CN=Puppet CA: host005]

Error: /File[/opt/puppetlabs/puppet/lib]: Could not evaluate: Could not
retrieve file metadata for puppet:///plugins: SSL_connect returned=1
errno=0 state=error: certificate verify failed: [self signed certificate in
certificate chain for /CN=Puppet CA: host005]

Error: Could not retrieve catalog from remote server: SSL_connect
returned=1 errno=0 state=error: certificate verify failed: [self signed
certificate in certificate chain for /CN=Puppet CA: host005]

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Error: Could not send report: SSL_connect returned=1 errno=0 state=error:
certificate verify failed: [self signed certificate in certificate chain
for /CN=Puppet CA: host005]
[root@node001 ~]#

Does anyone have an indication as to why this may be happening?

#2

Believe we've had some success.

When the Puppet master was created it has gone and created it's certificate:

[root@host005~]# puppet cert list --all

+ "host005" (SHA256)

And determined this to be self-signed by host005, being listed as the
issuer, using openssl x509 -text -noout -in
/etc/puppetlabs/puppet/ssl/certs/host005.pem.

After referring to the following resources over the past week:

··· - https://ask.puppet.com/question/15772/how-to-setup-multiple-puppet-masters-with-a-single-ca/ - https://serverfault.com/questions/644673/how-to-seperate-the-ca-server-from-master-basic-setup - https://blkperl.github.io/replace-puppet-ca.html

We:

  1. Revoked host005’s certificate
  2. Stopped all the Puppet services and httpd
  3. On host004, generated certificates for *host005 *(puppet cert
    generate host005    )
  4. Copied the contents of /etc/puppetlabs/puppet/ssl from *host004 *>
    host005
  5. Restarted *httpd *and all the Puppet services

After this we cleaned certs for node001; completed CSR with the Puppet
certificate authority (host004); was able to pull content from the Puppet
master (host005).

Still to validate with additional nodes and some further testing to verify
nothing in our playpen has gotten borked.

Hopefully this proves of some use to anyone else that encounters the issue.

On Tuesday, October 24, 2017 at 5:49:00 PM UTC+8, BJ wrote:

G’day everyone.

So far, have built out:

  • *host001 *- PostgreSQL
  • *host002 *- memcached
  • *host003 *- Foreman UI with Katello
  • *host004 *- Puppet certificate authority
  • *host005 *- Puppet compile master

Where:

  • *host003 *uses *host001 *for persistent storage, *host002 *for
    transient values (in the event we create more Foreman nodes)
  • *host004 *is connected to *host003 *using Smart Proxy
  • *host005 *is connected to *host003 *using Smart Proxy

If we take a new host - *node001 *- and modify puppet.conf to read like

  • [main]*

  • ca_server = host004*

  • [agent]*

  • server = host005*

And execute puppet agent --test --verbose, we can successfully begin a
certificate signing request.

After we sign the request on host004, we execute puppet agent --test
–verbose from node001 again, but encounter the following:

[root@node001 ~]# puppet agent --test --verbose
Warning: Unable to fetch my node definition, but the agent run will
continue:

Warning: SSL_connect returned=1 errno=0 state=error: certificate verify
failed: [self signed certificate in certificate chain for /CN=Puppet CA:
host005]

Info: Retrieving pluginfacts

Error: /File[/opt/puppetlabs/puppet/facts.d]: Failed to generate
additional resources using ‘eval_generate’: SSL_connect returned=1 errno=0
state=error: certificate verify failed: [self signed certificate in
certificate chain for /CN=Puppet CA: host005]

Error: /File[/opt/puppetlabs/puppet/facts.d]: Could not evaluate: Could
not retrieve file metadata for puppet:///pluginfacts: SSL_connect
returned=1 errno=0 state=error: certificate verify failed: [self signed
certificate in certificate chain for /CN=Puppet CA: host005]

Info: Retrieving plugin

Error: /File[/opt/puppetlabs/puppet/lib]: Failed to generate additional
resources using ‘eval_generate’: SSL_connect returned=1 errno=0
state=error: certificate verify failed: [self signed certificate in
certificate chain for /CN=Puppet CA: host005]

Error: /File[/opt/puppetlabs/puppet/lib]: Could not evaluate: Could not
retrieve file metadata for puppet:///plugins: SSL_connect returned=1
errno=0 state=error: certificate verify failed: [self signed certificate in
certificate chain for /CN=Puppet CA: host005]

Error: Could not retrieve catalog from remote server: SSL_connect
returned=1 errno=0 state=error: certificate verify failed: [self signed
certificate in certificate chain for /CN=Puppet CA: host005]

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Error: Could not send report: SSL_connect returned=1 errno=0 state=error:
certificate verify failed: [self signed certificate in certificate chain
for /CN=Puppet CA: host005]
[root@node001 ~]#

Does anyone have an indication as to why this may be happening?