SSL Certificate verify failed dynflow + smart proxy

Support
, ,
1

Problem: Foreman UI is using a custom certificate than the rest of the features/plugins, however I’m unable to execute remote jobs since the job fails due to a ssl error, I’m not exactly sure where the problem begins since this error is showing up at the proxy logs and the dynflow logs:

==> /var/log/foreman-proxy/proxy.log <==
2020-10-27T11:35:15 2cbbb5af [I] Started GET /dynflow/tasks/count state=running
2020-10-27T11:35:15 2cbbb5af [W] Error processing request '2cbbb5af-a659-4356-a18c-8e2cd24c1690
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
/usr/share/ruby/net/http.rb:921:in `connect'
/usr/share/ruby/net/http.rb:921:in `block in connect'
/usr/share/ruby/timeout.rb:52:in `timeout'
/usr/share/ruby/net/http.rb:921:in `connect'
/usr/share/ruby/net/http.rb:862:in `do_start'
/usr/share/ruby/net/http.rb:851:in `start'
/usr/share/ruby/net/http.rb:1373:in `request'
/usr/share/foreman-proxy/lib/proxy/request.rb:49:in `send_request'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/callback.rb:23:in `relay'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/callback.rb:29:in `relay'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/helpers.rb:5:in `relay_request'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/api.rb:62:in `block in <class:Api>'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `[]'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `catch'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `each'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:98:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/nulllogger.rb:9:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/head.rb:13:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/show_exceptions.rb:25:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `block in call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in `synchronize'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:66:in `block in call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `each'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:66:in `block in call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `each'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/builder.rb:153:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/handler/webrick.rb:88:in `service'
/usr/share/ruby/webrick/httpserver.rb:138:in `service'
/usr/share/ruby/webrick/httpserver.rb:94:in `run'
/usr/share/ruby/webrick/server.rb:295:in `block in start_thread'
/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `call'
/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-10-27T11:35:15 2cbbb5af [I] Finished GET /dynflow/tasks/count with 500 (14.94 ms)

==> /var/log/foreman-proxy/smart_proxy_dynflow_core.log <==
SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca (OpenSSL::SSL::SSLError)
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:299:in `accept'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:299:in `block (2 levels) in start_thread'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/utils.rb:263:in `timeout'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:297:in `block in start_thread'

Everything else works as expected, package synchronization, adding new content hosts, etc, it only fails when I want to execute a remote job to install x package on the destination content host or just issue a uname -r command I have tried the post here and the bug mentioned in the same post and the issue still persists. I have also tried to understand who connects to dynflow and to where dynflow connects to using this diagram but im still not sure about the connections for dynflow that would affect the ssl verification

Expected outcome: Be able to execute remote jobs

Foreman and Proxy versions: 1.24.3

Foreman and Proxy plugin versions: katello 3.14

Distribution and version: CentOS 7

Other relevant data:

dynflow settings:

# URL of the foreman, used for reporting back
:foreman_url: https://katello.domain.com

# SSL settings for client authentication against foreman.
:foreman_ssl_ca: FOREMAN UI CA
#:foreman_ssl_certificate: ssl/certs/fqdn.pem
#:foreman_ssl_private_key: ssl/private_keys/fqdn.pem

# Listen on address
:listen: 0.0.0.0

# Listen on port
:port: 8008

:use_https: true
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/fqdn.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/fqdn.pem

foreman-proxy settings:

:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/fqdn.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/fqdn.pem

# Endpoint for reverse communication
:foreman_url: https://katello.domain.com

# SSL settings for client authentication against Foreman. If undefined, the values
# from general SSL options are used instead. Mainly useful when Foreman uses
# different certificates for its web UI and for smart-proxy requests.
:foreman_ssl_ca: FOREMAN UI CA
#:foreman_ssl_cert: ssl/certs/fqdn.pem
#:foreman_ssl_key: ssl/private_keys/fqdn.pem
2

If you haven’t tried yet, make sure that the host system accepts the Puppet CA as a trusted certificate authority. At least from what I know, the smart proxy will use certs issued by the Puppet CA for all of it’s services per default, which means that the host system (which is running foreman) will not be able to access dynflow locally.

To check what cert is used is using try:

openssl s_client -showcerts -servername theforeman.example.com -connect theforeman.example.com:8008 2>/dev/null | openssl x509 -inform pem -noout -text

You should see the issuer somwhere at the beginning of the ouput, eg. something like this:

Certificate:
    Data:
        Version: %
        Serial Number: %
    Signature Algorithm: %
        Issuer: CN=Puppet CA: theforeman.example.com

If it says someting along the lines of the Puppet CA being used, try the solution from my post here, eg. symlinking the certs found under

/etc/puppetlabs/puppet/ssl/ca

into

/etc/pki/ca-trust/source/anchors/

and then running:

update-ca-trust
1 Like
3

Thank you! I will give it a try in around some hours and report back my results.

4

So I tried what @jimkoen suggested, but the issue still persists with the same output in the proxy.log and the dynflow log

5

So, after further digging into it, I found out that the foreman proxy ( /usr/share/foreman-proxy/lib/proxy/request.rb on line 49 ) it specifies that it will use the foreman_ssl_ca cert to connect with other services, the certificate I had there was actually the custom ssl CA for the domain name, so I did a join of my custom CA + Puppet CA cert and updated the foreman_ssl config within the proxy settings file to use the newly created cert and it is now properly working.

/etc/foreman-proxy/settings.yml
:foreman_ssl_ca: /path/to/customCA+puppetCA.crt