Uploading facts/reports after Custom SSL for Foreman

[corvar]

Problem: After installing a custom ssl (roughly using the directions at: Foreman :: Replacing Foreman's web SSL certificate.) I had issues uploading reports and facts to Foreman.

Expected outcome: Follow instructions, everything works

Foreman and Proxy versions:
foreman:amd64/buster 1.24.2-1 uptodate
foreman-cli:all/buster 1.24.2-1 uptodate
foreman-debug:all/buster 1.24.2-1 uptodate
foreman-ec2:all/buster 1.24.2-1 uptodate
foreman-installer:all/buster 1.24.2-1 uptodate
foreman-postgresql:all/buster 1.24.2-1 uptodate
foreman-proxy:all/buster 1.24.2-1 uptodate
ruby-hammer-cli-foreman:all/buster 0.19.7-1 uptodate

Foreman and Proxy plugin versions:

Distribution and version:
Debian Buster 10.3

Other relevant data:
Logs:

 2020-03-31T09:41:51.635-05:00 WARN  [qtp563245908-39] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During fact upload occured an exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) /etc/puppetlabs/puppet/node.rb:414: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

2020-03-31T09:41:56.278-05:00 INFO  [clojure-agent-send-pool-0] [puppetserver] Puppet Puppet settings initialized; run mode: master
2020-03-31T09:41:58.220-05:00 INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-agents] Finished creating JRubyInstance 3 of 4
2020-03-31T09:41:58.221-05:00 INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-internal] Creating JRubyInstance with id 4.
2020-03-31T09:41:58.975-05:00 WARN  [qtp563245908-41] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During fact upload occured an exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
/etc/puppetlabs/puppet/node.rb:414: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
2020-03-31T09:42:04.558-05:00 INFO  [clojure-agent-send-pool-0] [puppetserver] Puppet Puppet settings initialized; run mode: master
2020-03-31T09:42:07.335-05:00 INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-agents] Finished creating JRubyInstance 4 of 4
2020-03-31T09:42:13.176-05:00 INFO  [qtp563245908-41] [puppetserver] Puppet Compiled catalog for per415.theonering.net in environment production in 14.00 seconds
2020-03-31T09:42:24.073-05:00 ERROR [qtp563245908-42] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://fqdn/api/config_reports: No message available
["org/jruby/ext/openssl/SSLContext.java:402:in `setup'", "org/jruby/ext/openssl/SSLSocket.java:189:in `initialize'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:977:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'", "org/jruby/RubyArray.java:1800:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:290:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'", "org/jruby/RubyArray.java:1800:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `block in handleRequest'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:290:in `override'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:63:in `handleRequest'"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:75:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'
org/jruby/RubyArray.java:1800:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:290:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1800:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:290:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:63:in `handleRequest'
2020-03-31T09:58:25.819-05:00 WARN  [qtp563245908-39] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /etc/puppetlabs/puppet/node.rb:414: warning: constant ::TimeoutError is deprecated
SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

I was able to get reports to upload by specifying the servers web certificate in foreman.yaml :ssl_ca
I was able to get reports and facts to upload by commenting out :ssl_ca but that is non-ideal.

[corvar]

The specifics of exactly what I have done

### enable mod_rewrite
a2enmod rewrite
### Add rule rewrite rule to allow certificate validation
### I added the following block in /etc/apache2/sites-available/05-foreman.conf between ServerAlias and ## Custom fragement
  <IfModule mod_rewrite.c>
    RewriteEngine on
    RewriteCond /var/www/html$1 -f
    RewriteRule ^(/.well-known/.*)$ /var/www/html/$1 [L]
  </IfModule>
### Install certbot
apt-get install certbot
### Restart Apache
systemctl restart apache2
### If you have firewalled off http/https, you need to open up to the world.
iptables -I INPUT 5  -p tcp -m multiport --dports 80,443 -j ACCEPT
### Run certbot in certonly mode
certbot certonly --webroot --webroot-path /var/www/html -d fqdn_hostname
### Your certificates should now be in /etc/letsencrypt/live/fqdn_hostname
### Edit /etc/apache2/sites-available/05-foreman-ssl.conf.  I commented out my old lines, and added new lines so I could go back
#  SSLCertificateFile      "/etc/puppetlabs/puppet/ssl/certs/fqdn_hostname.pem"
#  SSLCertificateKeyFile   "/etc/puppetlabs/puppet/ssl/private_keys/fqdn_hostname.pem"
#  SSLCertificateChainFile "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
  SSLCertificateFile      "/etc/letsencrypt/live/fqdn_hostname/cert.pem"
  SSLCertificateKeyFile   "/etc/letsencrypt/live/fqdn_hostname/privkey.pem"
  SSLCertificateChainFile "/etc/letsencrypt/live/fqdn_hostname/fullchain.pem"
### Edit /etc/puppetlabs/puppet/foreman.yaml
### Update ssl_ca to the fullchain.pem (eventually tried chain.pem and cert.pem and adjusted permissions on directory path)
### the line currently says :ssl_ca: "/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem" 
:ssl_ca: "/etc/letsencrypt/live/fqdn_hostname/fullchain.pem"
### Restart shit
systemctl restart apache2
systemctl restart puppetserver
###  Test test test

[Woefdram]

Looks a lot like the problem I had (Foreman can't connect to PuppetDB).

Debian Buster and an unmodified /etc/ssl/openssl.cnf could be the problem, try changing

CipherString = DEFAULT@SECLEVEL=2

to

CipherString = DEFAULT

[corvar]

I changed that setting in openssl.cnf, put back in the :ssl_ca directive in foreman.yaml, and still can upload reports but no go on the facts.

2020-03-31T14:37:10.240-05:00 WARN  [qtp1006430289-41] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During fact upload occured an exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get issuer certificate)
/etc/puppetlabs/puppet/node.rb:414: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get issuer certificate)

[Woefdram]

And what if you keep the ca-file out and change openssl.cnf? That’s what worked for me.

[corvar]

I didn’t need to change openssl.cnf, with just the commenting out of :ssl_ca, facts and reports both uploaded. Looking at the code in node.rb, if ssl_ca is not specified, it does not try to validate the SSL certificate, and SSL seems to work, it is something related to the signatures on the certificate, and what CA it is validating it against (or so it seems to me).

[corvar]

I knew my issues were all related to CA, and I ran into some not just on the fact/report upload side, but also related to smart_proxy. So I dug into things a bit more and found a post by @ekohl a couple of years back (but more recent than the advice to edit the config files) with an example of using the foreman-installer to do this.

Replacing self-signed certificates

Yes, I’ve been meaning to write a blog about this. I’ll leave placing the certificates in the right place as a users exercise (I used LetsEncrypt). You want to change the CA that’s used to verify as well. I was lazy and used the system CA bundle. You gain additional security against another CA issuing a cert by using the the exact certificate but IMHO that’s a very small chance and not a relevant attack vector for my personal infra. --foreman-server-ssl-cert /etc/letsencrypt/live/$HOSTNAME/cert…

So I took that and pointed it at my letsencrypt files, changed ca-bundle.crt to ca-certificates.crt and let fly.

This seems to have resolved all of my SSL related issues, but I believe the ce portions may not be what I should be doing for security (–puppet-server-foreman-ssl-ca /etc/ssl/certs/ca-certificates.crt --foreman-proxy-foreman-ssl-ca /etc/ssl/certs/ca-certificates.crt).

Is that essentially saying that any SSL usable on a browser on my system would be able to authenticate to foreman to upload facts, interact with the smart-proxy, etc? And if so, conceptually what should those ca files be that limits access appropriately?