STDERR: During fact upload occured an exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed

llyons · October 11, 2022, 10:32pm

Problem:

running on client puppet agent -t we are getting the following errors.

C:\ProgramData\PuppetLabs\puppet\etc>puppet agent -tv
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Could not load external node results for iabrw-hfweb02.olh.local: (<unknown>): Tried to load unspecified class: Puppet::Node::Facts
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node iabrw-hfweb02.olh.local: Could not load external node results for iabrw-hfweb02.olh.local: (<unknown>): Tried to load unspecified class: Puppet::Node::Facts
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

in the puppetserver.log we have this error.

2022-10-11T17:24:40.491-05:00 ERROR [qtp789209799-39] [puppetserver] Puppet Server Error: Failed when searching for node iabrw-hfweb02.olh.local: Failed to find iabrw-hfweb02.olh.local via exec: Execution of '/etc/puppetlabs/puppet/node.rb iabrw-hfweb02.olh.local' returned 1:
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:222:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:368:in `block in find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:365:in `find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:407:in `node_from_request'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:138:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:297:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1809:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:65:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:297:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `handleRequest'
2022-10-11T17:24:40.617-05:00 ERROR [qtp789209799-34] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://uabrl-fore02.olh.local/api/config_reports: Received fatal alert: unknown_ca
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:297:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:65:in `block in handleRequest'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:297:in `override'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `handleRequest'"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:75:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'
org/jruby/RubyArray.java:1809:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:297:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1809:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:65:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:297:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `handleRequest'

Expected outcome:

with theforeman / puppet up and running we would expect the client to have facts uploaded properly and have the node associated with config/host groups / classes have those applied

Foreman and Proxy versions:
foreman 3.3

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

We are getting the iabrw-hfweb02.olh.local.yaml in the folder /opt/puppetlabs/server/data/puppetserver/yaml/facts after doing the puppet agent -t

It seems like we are having some form of cert error but not sure how to check this or resolve.

any help would be appreciated.

llyons · October 11, 2022, 10:46pm

For some reason I cant edit the above to add more info so wanted to add more here.

in /etc/foreman-proxy/settings.yml we have these

:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem


and

:foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
:foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem

also in /etc/puppetlabs/puppet/foreman.yaml

url: "https://uabrl-fore02.olh.local"
:ssl_ca: "/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem"
:ssl_cert: "/etc/puppetlabs/puppet/ssl/certs/uabrl-fore02.olh.local.pem"
:ssl_key: "/etc/puppetlabs/puppet/ssl/private_keys/uabrl-fore02.olh.local.pem"
:puppetdir: "/opt/puppetlabs/server/data/puppetserver"
:puppetuser: "puppet"
:facts: true
:fact_extension: "yaml"
:timeout: 60
:report_timeout: 60
:report_retry_limit: 1
:threads: null

in /etc/httpd/conf.d/05-foreman-ssl.conf

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/pki/katello/certs/katello-apache.crt"
  SSLCertificateKeyFile   "/etc/pki/katello/private/katello-apache.key"
  SSLCertificateChainFile "/etc/pki/katello/certs/katello-server-ca.crt"
  SSLVerifyClient         optional
  SSLVerifyDepth          3
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt"
  SSLOptions +StdEnvVars +ExportCertData

llyons · October 12, 2022, 2:33pm

We where able to figure this out or get passed the error with some help from VendettaMike.

had to explicitly reference Katello certs in the foreman installer command.

foreman-installer --enable-foreman-plugin-puppet --enable-foreman-cli-puppet \
--foreman-proxy-puppet true --foreman-proxy-puppetca true \
--foreman-proxy-content-puppet true --enable-puppet --puppet-server true \
--puppet-server-foreman-ssl-ca "/etc/pki/katello/puppet/puppet_client_ca.crt" \
--puppet-server-foreman-ssl-cert "/etc/pki/katello/puppet/puppet_client.crt" \
--puppet-server-foreman-ssl-key "/etc/pki/katello/puppet/puppet_client.key"

gvde · October 12, 2022, 3:16pm

Which is how it’s supposed to be done according to the docs: Configuring Hosts Using Puppet