Problem:
When I run a remote execution task, it never completes and eventually fails. The task does actually execute on the remote system. In the logs I see:
OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
Expected outcome:
No SSL errors
Foreman and Proxy versions:
3.11.1
Foreman and Proxy plugin versions:
| foreman-tasks | 9.1.1 |
|---|---|
| foreman_ansible | 14.0.0 |
| foreman_bootdisk | 21.2.3 |
| foreman_dhcp_browser | 0.0.8 |
| foreman_discovery | 24.0.1 |
| foreman_git_templates | 2.0.0 |
| foreman_puppet | 6.3.0 |
| foreman_remote_execution | 13.1.0 |
| foreman_templates | 9.4.0 |
Distribution and version:
Rocky Linux 9.4
Other relevant data:
I’m at a high frustration level. I had a Foreman+Katello system that was working, and somehow the certs got screwed up. After about 20 hours of troubleshooting, I gave up and decided to build a Foreman-only box from scratch.
I set up a brand new box from Rocky Linux 9.4. I set up Lets Encrypt for certificates. I used this exact command to do the initial install:
foreman-installer \
-l INFO \
--foreman-server-ssl-cert /etc/letsencrypt/live/${HOSTNAME}/cert.pem \
--foreman-server-ssl-chain /etc/letsencrypt/live/${HOSTNAME}/chain.pem \
--foreman-server-ssl-key /etc/letsencrypt/live/${HOSTNAME}/privkey.pem \
--foreman-proxy-foreman-ssl-ca /etc/ssl/certs/ca-certificates.crt \
--puppet-server-foreman-ssl-ca /etc/ssl/certs/ca-certificates.crt
This is from How-To use Foreman with Let'sEncrypt. The server installed and worked.
Now I’m trying to get remote execution working. When I trigger the job/task I can see Foreman log in to the remote machine, but I see this in the logs:
2024-08-12T16:59:52 87c6cccd [E] <OpenSSL::SSL::SSLError> SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
Okay… what certificate is that, and what process is making the connection? Foreman-proxy is using the LE cert. Is there a setting somewhere or an installer command to specify a different trusted root? Is remote execution somehow expecting the Puppet certificate?
I don’t know why this is this hard. Of all the applications I’ve set up certificates for, Foreman is by far the most difficult to get running and keep running. I do appreciate that Foreman uses certificates by default and doesn’t just skip verification.
Logs from foreman-proxy
2024-08-12T16:59:03 35c3ea37 [I] Finished POST /dynflow/tasks/39fa4030-65aa-4543-8f20-7467eff9b5a5/cancel with 500 (2.57 ms)
2024-08-12T16:59:45 87c6cccd [I] Started GET /dynflow/tasks/count state=running
2024-08-12T16:59:45 87c6cccd [I] Finished GET /dynflow/tasks/count with 200 (1.8 ms)
2024-08-12T16:59:46 87c6cccd [I] Started POST /dynflow/tasks/launch
2024-08-12T16:59:46 87c6cccd [I] Finished POST /dynflow/tasks/launch with 200 (19.69 ms)
2024-08-12T16:59:52 87c6cccd [E] <OpenSSL::SSL::SSLError> SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
/usr/share/ruby/net/protocol.rb:46:in `connect_nonblock'
/usr/share/ruby/net/protocol.rb:46:in `ssl_socket_connect'
/usr/share/ruby/net/http.rb:1038:in `connect'
/usr/share/ruby/net/http.rb:970:in `do_start'
/usr/share/ruby/net/http.rb:959:in `start'
/usr/share/ruby/net/http.rb:1512:in `request'
/usr/share/foreman-proxy/lib/proxy/request.rb:48:in `send_request'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/callback.rb:15:in `callback'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/callback.rb:9:in `send_to_foreman_tasks'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/callback.rb:31:in `run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in `block in run'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:51:in `restore_current_request_id'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:33:in `run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `catch'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `catch'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:583:in `execute_run'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:304:in `execute'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/director.rb:70:in `execute'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/parallel/worker.rb:16:in `block in on_message'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/parallel/worker.rb:15:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/actor.rb:122:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/actor.rb:56:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-08-12T17:03:41 91489588 [I] Started GET /version
2024-08-12T17:03:41 91489588 [I] Finished GET /version with 200 (1.09 ms)
2024-08-12T17:09:45 87c6cccd [I] Started POST /dynflow/tasks/status
2024-08-12T17:09:45 87c6cccd [I] Finished POST /dynflow/tasks/status with 200 (1.67 ms)