Puppet reports suddenly stopped at same time for all hosts

Support
1

Problem:
I’ve been running Foreman for many years now. Suddenly, all of my hosts are shown as out of sync (all stopped around 36 hours ago) and no Puppet reports from after that time are listed.

But Puppet reports are being sent and Foreman is sending correct ENC data to Puppet.

When hosts call in the Foreman logs look correct:

2021-10-02T02:11:19 [I|app|4f05af6d] Started GET ""/node/*****?format=yml"" for 172.31.84.205 at 2021-10-02 02:11:19 +0000
2021-10-02T02:11:19 [I|app|4f05af6d] Processing by HostsController#externalNodes as YML
2021-10-02T02:11:19 [I|app|4f05af6d]   Parameters: {""name""=>""*****""}
2021-10-02T02:11:19 [I|app|4f05af6d]   Rendering text template
2021-10-02T02:11:19 [I|app|4f05af6d]   Rendered text template (Duration: 0.0ms | Allocations: 2)
2021-10-02T02:11:19 [I|app|4f05af6d] Completed 200 OK in 95ms (Views: 0.5ms | ActiveRecord: 39.6ms | Allocations: 28054)
2021-10-02T02:11:26 [I|app|6e8a9513] Started POST ""/api/hosts/facts"" for 172.31.84.205 at 2021-10-02 02:11:26 +0000
2021-10-02T02:11:26 [I|app|6e8a9513] Processing by Api::V2::HostsController#facts as JSON
2021-10-02T02:11:26 [I|app|6e8a9513]   Parameters: {""facts""=>""[FILTERED]"", ""name""=>""*****"", ""certname""=>""*****"", ""apiv""=>""v2"", ""host""=>{""certname""=>""*****"", ""name""=>""*****""}}
2021-10-02T02:11:26 [W|app|6e8a9513] Some subtrees exceeded 100 limit of facts, dropped 101 keys
2021-10-02T02:11:28 [I|app|6e8a9513] Import facts for '*****' completed. Added: 0, Updated: 17, Deleted 0 facts
2021-10-02T02:11:28 [I|app|6e8a9513] Completed 201 Created in 1465ms (Views: 2.7ms | ActiveRecord: 764.9ms | Allocations: 316626)
2021-10-02T02:11:28 [I|app|44dd3bb8] Started GET ""/node/*****?format=yml"" for 172.31.84.205 at 2021-10-02 02:11:28 +0000
2021-10-02T02:11:28 [I|app|44dd3bb8] Processing by HostsController#externalNodes as YML
2021-10-02T02:11:28 [I|app|44dd3bb8]   Parameters: {""name""=>""*****""}
2021-10-02T02:11:28 [I|app|44dd3bb8]   Rendering text template
2021-10-02T02:11:28 [I|app|44dd3bb8]   Rendered text template (Duration: 0.0ms | Allocations: 2)
2021-10-02T02:11:28 [I|app|44dd3bb8] Completed 200 OK in 84ms (Views: 0.6ms | ActiveRecord: 35.9ms | Allocations: 28054)

Expected outcome:
That Foreman continues to know the reports it is receiving.

Foreman and Proxy versions:
Foreman and proxies all 2.5.3

Foreman and Proxy plugin versions:
foreman_bootdisk 17.1.0
foreman_setup 6.0.0

Distribution and version:
Ubuntu 18.04

Other relevant data:
Both Foreman and Puppet are configured through Puppet. So I am pretty certain no configuration has changed.

But, Ubuntu’s unattended-upgrades did run during the time frame this issue happened (it should only apply security fixes) and made the following changes on the Foreman server:

Start-Date: 2021-09-29  14:04:26
Commandline: /usr/bin/unattended-upgrade
Upgrade: apache2-data:amd64 (2.4.29-1ubuntu4.17, 2.4.29-1ubuntu4.18), apache2-bin:amd64 (2.4.29-1ubuntu4.17, 2.4.29-1ubuntu4.18), apache2:amd64 (2.4.29-1ubuntu4.17, 2.4.29-1ubuntu4.18), apache2-utils:amd64 (2.4.29-1ubuntu4.17, 2.4.29-1ubuntu4.18)
End-Date: 2021-09-29  14:04:30

I imagine this could affect Foreman since it uses Apache (I think so?). I tried to downgrade back to 2.4.29-1ubuntu4.17 but that version doesn’t seem to exist:

root@foreman:~# apt-cache madison apache2
   apache2 | 2.4.29-1ubuntu4.18 | http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
   apache2 | 2.4.29-1ubuntu4.18 | http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
   apache2 | 2.4.29-1ubuntu4 | http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

Yes, I have rebooted both the Foreman and Puppet servers.

2

Check your puppetserver version - if you upgraded to Puppet7 (maybe without realizing it), that changes the default upload formats from YAML to JSON and if Foreman was configured for Puppet5 or 6, it won’t recognize them anymore.

Check your /etc/puppetlabs/puppet/foreman.yaml and see what the fact_extension is. If you’re running puppetserver 7+, it should be “json”.

1 Like
3

Maybe the update modified some apache configuration needed by foreman. Running “foreman-installer” should reconfigure everything again. Verify, that apache has been restarted during the foreman-installer run or run “foreman-maintain service stop” and then “foreman-installer”.

1 Like
4

OK, I think I’m getting to the bottom of it. As I looked again at the log I sent above I realized nothing in there made sense for the Puppet report being sent. So I compared the 3 HTP requests in the log above to what I was seeing a week again and yes, the call to /api/config_reports is missing.

Tracking this down further I looked in the Puppetserver logs and noticed this:

2021-10-02T00:03:10.742Z ERROR [qtp188593319-57] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://****/api/config_reports: certificate verify failed

Browsers think there is nothing wrong with the certificate on my Foreman server but I do use Letsencrypt for it, and I have seen reports around the web of their old root cert expiring.

But for the life of me, so far I cannot find out how to get puppetserver to trust the new Letsencrypt cert.

So this is not a Foreman problem in the end.

Thanks for your suggestions.

5

There is a tutorial on how to use let‘s encrypt with foreman in the tutorials section. Maybe you want to check that out…

6

OK, finally managed to solve it. Currently Puppet can’t handle accessing standard Letsencrypt certificates. This I think is the underlying issue: https://github.com/jruby/jruby-openssl/issues/236

I tried deleting the expired cert from Puppet’s keystore but that made no difference. In the end I had to add an option to how the letsencrypt cert for Foreman is generated: --preferred-chain="ISRG Root X1". Annoyingly this option is not available in python3-certbot-apache on Ubuntu 18.04 so I had to install the certbot snap which you can’t manage through Puppet… blah blah sob story.

Not how I wanted to spend half of my Saturday.

1 Like