Foreman / Smart Proxy SSL error after Puppet Server upgrade

Support
1

TLDR;
Problem:
Reports not uploaded to Foreman via the Smart Proxy (SSL error) after an upgrade to Puppet Server 7.9.x (from 7.8.x). Nothing was changed in the Hiera config files other then the Puppet Server package version.

Expected outcome:
Reports available in Foreman, as per usual.

Foreman and Proxy versions:
Foreman 3.4 / Puppet Server 7.9 / Puppet Agent 7.21

Foreman and Proxy plugin versions:
N/A

Distribution and version:
Debian 11

Other relevant data:
Ask if you need anything, too much to add straight in.

Long version :
Hello everyone !

We are having a couple of trouble with our Foreman / Puppet installation and after several weeks of forensics, we are unable to pinpoint the source of our problems.
So we’re turning to you, the Community, for some advices.

Our platform is built as follow :
We have 2 datacenters in which we have the exact same infrastructure.

  • 1 Foreman / Puppet CA (accessing a mutualized postgreSQL database on a network host)
  • 3 Puppet Servers / Foreman SmartProxies

2 load balanced virtual IPs, one for the Foreman / Puppet CA host (let’s call it foreman.acme.com) and another one for the SmartProxies (smartproxy.acme.com).

Datacenter 1 has the main Foreman / Puppet CA and 3 active SmartProxies (serving the hosts in that datacenter), and Datacenter 2 has the (cold) backup Foreman / Puppet CA and 3 active SmartProxies (serving the hosts in that datacenter).
So, a total of 8 servers.

The software versions are :

  • Foreman 3.4 branch, latest.
  • Puppet 7 branch, version 7.9.
    I can provide the version of other modules/packages if requested.
    We “auto” deployed the entire infrastructure using Puppet, not using the foreman installer.

Our problem :
1/ Hosts reports no longer being uploaded to the Puppet Master with version 7.9 (SSL error)
We’ve tried to upgrade the Puppet Servers to 7.9 (we’re currently using Puppet Servers 7.8).
But when doing so all the hosts running on the upgraded SPs are not reporting correctly.
A SSL error can be seen in the servers’ logs but we are at a complete loss about why.
Downgrading the servers to version 7.8 restore the reporting and new reports are then available on Foreman.
From what we’ve seen, it has to do with the HTTPS check when the SPs are trying to upload the report but we’re unable to find which setting (or settings) have changed between the version 7.8 and 7.9.
All the config files stayed the same so why ?

Also…

2/ We are experiencing random CPU spikes (as it seems) on the SmartProxies.
One or two of them suddenly skyrocket to 100% CPU usage without anything in the error, puppetserver, smartproxy logs.
We have set up a cron job to automatically restart the SPs every morning, but we still get those random 100% CPU usage spikes during the day.
I must add that we are currently in the process of migrating hosts from our old Foreman / Puppet platform to this new platform and adding roughly 100 hosts a day.
I can provide sample of our configuration files for the SPs if needed.

If you have any question or need any configuration files, please let us know.

Best regards,
Alban.

2

Here’s the error message that we get :

ERROR [qtp129146556-56] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://foreman.acme.com//api/config_reports: certificate verify failed
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1018:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:934:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:923:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1481:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'", "org/jruby/RubyArray.java:1865:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:181:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `block in call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:52:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:17:in `block in wrap'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'", "org/jruby/RubyArray.java:1865:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:69:in `block in with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:65:in `with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:92:in `respond_to_errors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:84:in `process'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:75:in `process'
3

So we did some more tests …
We’ve upgraded one of the smart proxy to Puppet Server version 7.9.3 while leaving Foreman (3.4.1) and the other SPs as is (Puppet Server 7.8.0).

As expected, all the hosts running on that smart proxy were not reporting correctly. And we could see the SSL error in the logs.

And we’ve narrowed down our problem a little more …

The Puppet Server version 7.8 is using jruby-openssl (default: 0.13.0 java) while the Puppet Server version 7.9 is using jruby-openssl (default: 0.14.0 java).

Since that gem is a “default” gem we can’t do much about it.

We decided to test “shunting” the SSL verification in Puppet Server 7.9 to see if the reports were available again. And they were. But that’s an ugly solution and does not meet our Quality standards.

What we dit :
Edit the file /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb line 56 and replace VERIFY_PEER by VERIFY_NONE, the reports are uploaded again and visible in Foreman. Again deactivating SSL verification is not something I advise.

We’re planning to do some more tests about our SSL certificates and will post additional informations if useful.

4

Correction : default: 0.10.5 java

6

Looking through the GitHub Tag list, we found that on the 0.11.0 tag you could use an argument to add to /etc/default/puppetserver JAVA_ARGS parameter.

JAVA_ARGS="-Xms2G -Xmx2G -Djruby.log -Djruby.openssl.x509.store.verify=legacy "

7

Before hacking files, I would rather check the certificates first.

Check the file /etc/puppetlabs/puppet/foreman.yaml . It contains the setting for ssl_ca which is used for certificate verification, e.g.

...
:ssl_ca: "/etc/pki/katello/puppet/puppet_client_ca.crt"
...

Then check if it’s the correct certificate:

$ openssl s_client -connect foreman.example.com:443 -verify 3 -no-CApath -verify_return_error -CAfile /etc/pki/katello/puppet/puppet_client_ca.crt

It should connect successfully:

...
    Verify return code: 0 (ok)
...
8

We’re not using katello, but I get your point and will check that as soon as I will have the chance. :+1:

9

So we’ve tested, and yes. Same return code 0.
At least with the -Djruby.openssl.x509.store.verify=legacy option for the JVM.

That option seem to have resolved our problem, at least until we identify how to upgrade those gems. But the option was found on the gem owner GitHub repo and was put there for the specific problem we had.

Thank you for taking some time to offer some answers !