Problem: Foreman UI is using a custom certificate than the rest of the features/plugins, however I’m unable to execute remote jobs since the job fails due to a ssl error, I’m not exactly sure where the problem begins since this error is showing up at the proxy logs and the dynflow logs:
==> /var/log/foreman-proxy/proxy.log <==
2020-10-27T11:35:15 2cbbb5af [I] Started GET /dynflow/tasks/count state=running
2020-10-27T11:35:15 2cbbb5af [W] Error processing request '2cbbb5af-a659-4356-a18c-8e2cd24c1690
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
/usr/share/ruby/net/http.rb:921:in `connect'
/usr/share/ruby/net/http.rb:921:in `block in connect'
/usr/share/ruby/timeout.rb:52:in `timeout'
/usr/share/ruby/net/http.rb:921:in `connect'
/usr/share/ruby/net/http.rb:862:in `do_start'
/usr/share/ruby/net/http.rb:851:in `start'
/usr/share/ruby/net/http.rb:1373:in `request'
/usr/share/foreman-proxy/lib/proxy/request.rb:49:in `send_request'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/callback.rb:23:in `relay'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/callback.rb:29:in `relay'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/helpers.rb:5:in `relay_request'
/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/api.rb:62:in `block in <class:Api>'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `[]'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `catch'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `each'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:98:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/nulllogger.rb:9:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/head.rb:13:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/show_exceptions.rb:25:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `block in call'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in `synchronize'
/usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:66:in `block in call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `each'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:66:in `block in call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `each'
/usr/share/gems/gems/rack-1.6.12/lib/rack/urlmap.rb:50:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/builder.rb:153:in `call'
/usr/share/gems/gems/rack-1.6.12/lib/rack/handler/webrick.rb:88:in `service'
/usr/share/ruby/webrick/httpserver.rb:138:in `service'
/usr/share/ruby/webrick/httpserver.rb:94:in `run'
/usr/share/ruby/webrick/server.rb:295:in `block in start_thread'
/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `call'
/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-10-27T11:35:15 2cbbb5af [I] Finished GET /dynflow/tasks/count with 500 (14.94 ms)
==> /var/log/foreman-proxy/smart_proxy_dynflow_core.log <==
SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca (OpenSSL::SSL::SSLError)
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:299:in `accept'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:299:in `block (2 levels) in start_thread'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/utils.rb:263:in `timeout'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:297:in `block in start_thread'
Everything else works as expected, package synchronization, adding new content hosts, etc, it only fails when I want to execute a remote job to install x package on the destination content host or just issue a uname -r command I have tried the post here and the bug mentioned in the same post and the issue still persists. I have also tried to understand who connects to dynflow and to where dynflow connects to using this diagram but im still not sure about the connections for dynflow that would affect the ssl verification
Expected outcome: Be able to execute remote jobs
Foreman and Proxy versions: 1.24.3
Foreman and Proxy plugin versions: katello 3.14
Distribution and version: CentOS 7
Other relevant data:
dynflow settings:
# URL of the foreman, used for reporting back
:foreman_url: https://katello.domain.com
# SSL settings for client authentication against foreman.
:foreman_ssl_ca: FOREMAN UI CA
#:foreman_ssl_certificate: ssl/certs/fqdn.pem
#:foreman_ssl_private_key: ssl/private_keys/fqdn.pem
# Listen on address
:listen: 0.0.0.0
# Listen on port
:port: 8008
:use_https: true
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/fqdn.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/fqdn.pem
foreman-proxy settings:
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/fqdn.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/fqdn.pem
# Endpoint for reverse communication
:foreman_url: https://katello.domain.com
# SSL settings for client authentication against Foreman. If undefined, the values
# from general SSL options are used instead. Mainly useful when Foreman uses
# different certificates for its web UI and for smart-proxy requests.
:foreman_ssl_ca: FOREMAN UI CA
#:foreman_ssl_cert: ssl/certs/fqdn.pem
#:foreman_ssl_key: ssl/private_keys/fqdn.pem