Problem:
Hello! I am trying to set up remote execution, however each time I run a job, the job fails with the following error message in the foreman web ui:
Failed to initialize: RuntimeError - The only applicable proxy theforeman.fbi.h-da.de is down
To explain, we’re using separate certs for the puppet CA and web ui. As far as I know, both proxy and dynflow are configured to use the puppet CA’s certificate, and the puppet CA cert should be propagated to the individual hosts during the preseeding / provisioning phase (it should, shouldn’t it?).
This bug report mentions a similar issue to mine and suggests changing the cert used by dynflow and to run remote jobs to the one used in the web UI. I have not yet attempted this, as I don’t understand the full impact of this change yet.
The important bit from proxy.conf regarding the SSL error when the job ran:
2020-10-13T15:19:11 19f509de [W] Error processing request '19f509de-7e05-462f-9f00-036c9991de41: <OpenSSL::SSL::SSLError>: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
The rest of the log entry in proxy.conf looks like this (I anonymized the IPv6 address):
2020-10-13T15:19:11 [D] accept: 2001::::21a:42782
2020-10-13T15:19:11 [D] Rack::Handler::WEBrick is invoked.
2020-10-13T15:19:11 19f509de [I] Started GET /dynflow/tasks/count state=running
2020-10-13T15:19:11 19f509de [D] verifying remote client foreman.myorg.com (based on SSL_CLIENT_CERT) against trusted_hosts ["foreman.myorg.com"]
2020-10-13T15:19:11 19f509de [D] Proxy request from foreman.myorg.com:8443/dynflow/tasks/count to https://foreman.myorg.com:8008/tasks/count
2020-10-13T15:19:11 19f509de [W] Error processing request '19f509de-7e05-462f-9f00-036c9991de41: <OpenSSL::SSL::SSLError>: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in `connect_nonblock'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in `ssl_socket_connect'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:985:in `connect'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:920:in `do_start'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:909:in `start'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1458:in `request'
/usr/share/foreman-proxy/lib/proxy/request.rb:48:in `send_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/callback.rb:23:in `relay'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/callback.rb:29:in `relay'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/helpers.rb:5:in `relay_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.2.4/lib/smart_proxy_dynflow/api.rb:62:in `block in <class:Api>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:in `block in compile!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in `block (3 levels) in route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1011:in `route_eval'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in `block (2 levels) in route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1040:in `block in process_route'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:in `process_route'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:990:in `block in route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:in `route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1097:in `block in dispatch!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1094:in `dispatch!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in `block in call!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in `call!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:913:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:103:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/xss_header.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/path_traversal.rb:16:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/json_csrf.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/null_logger.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/show_exceptions.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:194:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1958:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1729:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/builder.rb:244:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/handler/webrick.rb:95:in `service'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-10-13T15:19:11 19f509de [I] Finished GET /dynflow/tasks/count with 500 (20.6 ms)
2020-10-13T15:19:11 19f509de [D] close: 2001::::21a:42782
Expected outcome:
The remote execution job should run fine, as I think I have everything setup correctly.
Foreman and Proxy versions:
2.1.1
Foreman and Proxy plugin versions:
I assume the latest for 2.1.1
Puppet on the client reports 6.18.0,
Puppet on the server 6.12.1
Distribution and version:
CentOS 7.8.2003
Other relevant data: