Remote execution job is pending status

[cps86]

Hello

@ aruzicka
I think there’s a bug in foreman which is causing this.

I installed foreman (NOT katello) using the regular self-signed by foreman/puppet certs. I have not provided any cert-related paramters into the foreman installer. The command line which I used to install foreman was:

foreman-installer --foreman-db-host=myexternaldatabase.infra.com --foreman-db-username=foreman --foreman-db-password=‘mydatabasepassword’ --foreman-initial-admin-password myadminpassword

So, foreman and foreman-proxy are running on the same host.
I can’t see any errors for the smart proxy in the Infrastrucutre/Smart Proxy tab.
I installed remote execution plugin and I am able to execute tasks on a remote hosts… but I have a exactly the same problem as @brookst .

I configured 2 simple tasks which simply print a message on the screen. Regardless whether it’s a ansible playbook or ssh command it always almost immediately ends with “Exit status: 0”, like following:

PLAY RECAP *********************************************************************
23:
myhost.mycompany.net : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
24:
Exit status: 0

So it was executed correctly as I can see stdout, but I need to wait around 10 minutes to resolve the job status.
In the foreman smart-proxy log there’s following:

2024-12-16T14:53:30 c6273546 [E] OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed (self-signed certificate in certificate chain)
/usr/share/ruby/net/protocol.rb:46:in connect_nonblock' /usr/share/ruby/net/protocol.rb:46:in ssl_socket_connect’
/usr/share/ruby/net/http.rb:1038:in connect' /usr/share/ruby/net/http.rb:970:in do_start’
/usr/share/ruby/net/http.rb:959:in start' /usr/share/ruby/net/http.rb:1512:in request’
/usr/share/foreman-proxy/lib/proxy/request.rb:48:in send_request' /usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/callback.rb:15:in callback’
/usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/action/batch_callback.rb:16:in run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:590:in block (3 levels) in execute_run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:29:in with_progress_calculation' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:15:in run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in pass' /usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in block in run’
/usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:51:in restore_current_request_id' /usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in pass' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/world.rb:31:in execute’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:589:in block (2 levels) in execute_run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in catch’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in block in execute_run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in block in with_error_handling’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in catch' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in with_error_handling’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:583:in execute_run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:304:in execute’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in with_meta_calculation’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/director.rb:70:in execute’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/parallel/worker.rb:16:in block in on_message' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors.rb:18:in run_user_code’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/parallel/worker.rb:15:in on_message' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in on_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/actor.rb:122:in on_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in on_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/actor.rb:56:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in process_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in process_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in block in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in block (2 levels) in schedule_execution' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in block in synchronize’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in synchronize' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in synchronize’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in block in schedule_execution' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in call’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in work' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in block in call_job’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in run_task' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in block (3 levels) in create_worker’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in loop' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in block (2 levels) in create_worker’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in catch' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in block in create_worker’
/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

I tried to do the trick as @tomzellner in:

Foreman-Proxy - certificate verify failed (self signed certificate in certificate chain)

I have just installed a foreman content proxy on EL9 following the docs Installing a Smart Proxy Server 3.11 on Enterprise Linux with custom certificates. It works fine. No errors registering. Running on the main server: # foreman-proxy-certs-generate --foreman-proxy-fqdn foreman9-content.example.com \ --certs-tar /root/foreman9-content.example.com-certs.tar \ --server-cert /root/certs/foreman9-content/2024/foreman9-content_cert.cer \ --server-key /root/certs/foreman…

So I changed 64th line in the following file:

/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb

:ca_file => “/etc/puppetlabs/puppet/ssl/certs/ca.pem”

but it didn’t help.
My foreman proxy and foreman config (I HAVE NOT CHANGED ANYTHING AFTER INSTALLATION!):

Foreman:

:unattended: true
:require_ssl: true

:oauth_active: true
:oauth_map_users: false
:oauth_consumer_key: stWw3ErnxNGuUENTAiS3PKUf56rRUCzF
:oauth_consumer_secret: dYM8XeSBHwZnxhVF3TyxgN3cgCduMKCq

Websockets

:websockets_encrypt: true
:websockets_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/myservername.infra.net.pem
:websockets_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/myservername.infra.net.pem

SSL-settings

:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/myservername.infra.net.pem
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_priv_key: /etc/puppetlabs/puppet/ssl/private_keys/myservername.infra.net.pem

Foreman Proxy:

:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/myservername.infra.net.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/myservername.infra.net.pem

#:foreman_ssl_ca: ssl/certs/ca.pem
#:foreman_ssl_cert: ssl/certs/fqdn.pem
#:foreman_ssl_key: ssl/private_keys/fqdn.pem

I think this bug was introducted after 3.8 release because I didn’t see this behaviour over there.