Puppet Runs Fail After Replacing CA

Support
#1

Problem:
We were in the process of migrating from a PE install to a Foreman all in one install. Everything is migrated over save the CA, still on the old PE instance. That certificate expired and everything broke. I stood up a Puppetca foreman proxy and regenerated the CA. Currently most things appear to be working. After getting the new certs agents are able to complete their puppet runs, I can build hosts, etc. However, the puppet agent on the puppet master fails with a 403 forbidden error. Additionally, the PuppetCA doesn’t show up in the Smart Proxy list on Foreman. I’m able to sign new certs from the command line though, so I’m reasonably certain it is working.

I’ve been through quite a bit of trial and error trying to get the certs replaced and I believe I’m still missing a few pieces.
Expected outcome:

Foreman and Proxy versions:
Foreman 1.22.2, proxies are also 1.22.2

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: 403 "Forbidden"
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': 403 "Forbidden"
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: 403 "Forbidden"
Info: Retrieving plugin
Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': 403 "Forbidden"
Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: 403 "Forbidden"
Info: Loading facts
Error: Could not retrieve catalog from remote server: 403 "Forbidden"
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: 403 "Forbidden"
#2

Continuing to troubleshoot this, it appears that the puppet master does not trust itself. Client requests from other boxes work just fine, but anything the puppet master tries to do locally fails. I’ve got more 403’s in the puppetserver.log:

2020-05-14 13:50:44,935 ERROR [qtp734434868-26] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://<hostname>/api/config_reports: Received fatal alert: unknown_ca

2020-05-14 13:58:37,219 ERROR [qtp734434868-96] [p.t.a.rules] Forbidden request: <ip> access to /puppet/v3/node/<hostname> (method :get) (authenticated: false) denied by rule 'puppetlabs node'.
2020-05-14 13:58:37,314 ERROR [qtp734434868-43] [p.t.a.rules] Forbidden request: <ip> access to /puppet/v3/file_metadatas/pluginfacts (method :get) (authenticated: false) denied by rule 'puppetlabs file metadata'.
#3

Ok, I think I’ve resolved this. Turns out my http proxy was grabbing puppet requests. Once I added the puppet master hostname into the no_proxy environment variable it began to work.

I’ve still got a trust issue between foreman and puppet however, I’ll create a new topic for that.

1 Like