Upgraded to Foreman 2.5.4, now `Puppet Report processor failed: Could not send report to Foreman`

Problem:

On Tuesday, I upgraded from Foreman 2.3 → 2.4 → 2.5.4.

Today, I noticed that no reports are showing up in Foreman. Puppet agents are communicating with the Puppet Server just fine, and we are able to make configuration changes, etc.

Long story short, the cause seems to be this from production.log:

2021-11-04 13:04:31,482 ERROR [qtp161156004-20] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://foreman.example.org/api/config_reports: Received fatal alert: handshake_failure

I’m sure this is a TLS failure, but everything seems to look okay to me. How can I troubleshoot & verify that TLS is working between Puppet and Foreman? Is there a Foreman guide for troubleshooting TLS errors?

Expected outcome:

I expect the report processor to return success in the log files.

Foreman and Proxy versions:

2.5.4

Proxies:

  • BMC
  • DHCP
  • Discovery
  • HTTPBoot
  • Logs
  • Puppet
  • Puppet CA
  • Registration
  • TFTP

Foreman and Proxy plugin versions:

2.5.4

Distribution and version:

Ubuntu 18.04.6

Other relevant data:

root@foreman:/var/log/puppetlabs/puppetserver# tail -1000f puppetserver.log |grep ERROR
...
2021-11-04 13:04:31,482 ERROR [qtp161156004-20] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://foreman.example.org/api/config_reports: Received fatal alert: handshake_failure
["org/jruby/ext/openssl/SSLSocket.java:217:in `connect'", "/opt/puppetlabs/server/apps/puppetserver/jruby-1_7.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:800:in `connect'", "org/jruby/ext/timeout/Timeout.java:115:in `timeout'", "/opt/puppetlabs/server/apps/puppetserver/jruby1_7.jar!/METAINF/jruby.home/lib/ruby/1.9/net/http.rb:800:in `connect'", "/opt/puppetlabs/server/apps/puppetserver/jruby-1_7.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:756:in `do_start'", "/opt/puppetlabs/server/apps/puppetserver/jruby-1_7.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:745:in `start'", "/opt/puppetlabs/server/apps/puppetserver/jruby-1_7.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:1293:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:53:in `processors'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `sa
ve'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:289:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/ indirected_routes.rb:192:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:263:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'", "org/jru
by/RubyArray.java:1613:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:64:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'",
 "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:62:in `process'", "file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'", "Puppet$$Server$$Master_659347371.gen:13:in `handleRequest'", "request_handler_core.clj:273:in `invoke'", "jruby_request.clj:49:in `invoke'", "jruby_request.clj:34:in `invoke'", "request_handler_service.clj:47:in `handle_request'", "request_handler.clj:3:in `invoke'", "request_handler.clj:3:in `invoke'", "core.clj:2515:in `invoke'", "ring_middleware.clj:290:in `invoke'", "core.clj:170:in `invoke'", "core.clj:216:in `invoke'", "core.clj:47:in `invoke'", "core.clj:357:in `invoke'", "core.clj:53:in `invoke'", "ringutils.clj:83:in `invoke'", "master_core.clj:742:in `invoke'", "ring.cljc:25:in `invoke'", "ring.cljc:16:in `invoke'", "comidi.clj:245:in `invoke'", "http.clj:152:in `invoke'", 
"http.clj:152:in `invoke'", "http.clj:148:in `invoke'", "comidi.clj:332:in `invoke'", "jetty9_core.clj:434:in `invoke'", "normalized_uri_helpers.clj:74:in `invoke'"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:75:in `process'

The certificate is valid, as far as the host is concerned:

# openssl s_client -connect foreman.example.org:443 0>/dev/null 2>/dev/null | sed -ne '/---BEGIN CERTIFICATE---/,/---END CERTIFICATE---/p' | openssl x509 -noout -dates
notBefore=May  5 00:00:00 2020 GMT
notAfter=May  5 23:59:59 2022 GMT

But I’m not sure what Puppet or Foreman uses to do the same.

Stefan,

I ran across a similar problem recently. The puppet agent version on our clients was pretty old and there was an issue with the enabled cipher suites. We have just re-enabled the relevant suites under /etc/httpd/conf.d/ssl.conf temporarily until we are able to upgrade the clients soon.

1 Like

Thanks Cameron.

I solved this by adding a strong cipher list to /etc/foreman-installer/custom-hiera.yaml like so:

apache::mod::ssl::ssl_cipher: ECDH+AESGCM:ABCD:EFGHI:!RSA+AESGCM:!RSA+AES:!aNULL:!MD5:!DSS

This added the SSLCipherSuite to /etc/apache2/mods-available/ssl.conf like so:

SSLCipherSuite ECDH+AESGCM:ABCD:EFGHI:!RSA+AESGCM:!RSA+AES:!aNULL:!MD5:!DSS

Foreman has a strong cipher list at /usr/share/foreman-installer/config/foreman.hiera/security.yaml, but this list was being ignored and never made it into the Apache configuration.

1 Like

Hi,

I’ve encountered the exact same issue when going from 2.4 to 2.5.
We are running puppet server 5.3.16

I have installed httpd on CentOS and modifying the hiera.yml file didn’t help me out.
But it pointed me in the right direction.
I tried to change my httpd ssl.conf settings to the one you proposed but also this didn’t help.
Afterwards I checked the settings from version 2.4 and they are much more extented.

So i pasted these cyphers from version 2.4 and the reports started working again.

SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:!3DES:!DES:!DSS

1 Like

thank you @Stefan_Lasiewski it works for me :slight_smile:

1 Like

Hey, looks like we’re in the same boat. Your post over at Foreman We're sorry, but something went wrong - #16 by Stefan_Lasiewski just helped me! :smile:

1 Like

thats good @Stefan_Lasiewski so in this way we can help eachother :-9

1 Like