Problem:
Puppetserver is unable to send reports to the foreman api
Expected outcome:
Puppetserver is able to send reports
Foreman and Proxy versions:
2.3.3
Foreman and Proxy plugin versions:
2.3.3
Distribution and version:
Ubuntu 18.04
Other relevant data:
We first noticed that foreman was not displaying reports even though puppet was working correctly. After a bit of digging we found the culprit in puppetserver.log
2022-05-24T15:03:43.312-04:00 ERROR [qtp346761075-3017459] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://foreman1.example.net/api/config_reports: certificate verify failed
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:65:in `block in handleRequest'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `handleRequest'"]
I believe this is because when I installed foreman I was trying to get letsencrypt to work on the webui.
Looking at the history I think my install command was the following
foreman-installer --foreman-initial-organization='Acme' \
--foreman-initial-location='USA' \
--puppet-autosign-entries='*.example.net' \
--puppet-autosign-mode='0664' \
--puppet-splay=true \
--puppet-splaylimit=60s \
--foreman-server-ssl-cert /etc/letsencrypt/live/foreman1.example.net/cert.pem \
--foreman-server-ssl-key /etc/letsencrypt/live/foreman1.example.net/privkey.pem \
--foreman-server-ssl-chain /etc/letsencrypt/live/foreman1.example.net/chain.pem \
--puppet-server-foreman-ssl-ca /etc/ssl/certs/ca-certificates.crt \
--foreman-proxy-foreman-ssl-ca /etc/ssl/certs/ca-certificates.crt
This allows foreman to use letsencrypt certs, however they do not match the puppet ca cert, which I think is the problem.
I’ve tried adding the letsencrypt ca chain to the ca chain in /etc/puppetlabs/puppet/ssl/certs with no luck.
Anyone know how to solve this? Or if it even is solvable? I don’t think I’m the first one to run into this problem. https://community.theforeman.org/t/puppet-report-processor-failed/26097