Remote execution job is pending status

Support
1

Problem:
Remote execution job is pending status

Expected outcome:
It suppose to finish and needs to show in console as successfull

Foreman and Proxy versions:
foreman-proxy-3.6.1

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:
I am getting below error message while i am executing remote jobs from foreman console: (from proxy.log)

2023-10-22T07:35:06 965b5e9e [E] <OpenSSL::SSL::SSLError> SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
        /usr/share/ruby/net/protocol.rb:44:in `connect_nonblock'
        /usr/share/ruby/net/protocol.rb:44:in `ssl_socket_connect'
        /usr/share/ruby/net/http.rb:1009:in `connect'
        /usr/share/ruby/net/http.rb:943:in `do_start'
        /usr/share/ruby/net/http.rb:932:in `start'
        /usr/share/ruby/net/http.rb:1483:in `request'
        /usr/share/foreman-proxy/lib/proxy/request.rb:48:in `send_request'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/callback.rb:13:in `callback'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/callback.rb:7:in `send_to_foreman_tasks'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/callback.rb:28:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:15:in `block in run'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:15:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:69:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:122:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:56:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
        /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-10-22T07:35:14 965b5e9e [E] <Dynflow::Action::WithSubPlans::SubtaskFailedException> A sub task failed
2

Hi,
your smart proxy doesn’t trust Foreman’s certificate and so it fails when it tries to report the remote execution job results to Foreman. Are you using custom certificates?

3

Yes We are using Custom certificate and its generated via Venafi. How we can get rid of from this ?? Can you please let me know where will be the location of the certificate . Currently I am confused since we made many certification changes.

4

Well, that makes things somewhat hard. What is the timeline of those changes? Is the smart proxy mentioned in the first post the one co-located on the foreman server or is it an external one? Do you have katello enabled?

In general you should be able to take the certs you have from venafi, point the installer at them and the installer should take care of things. The relevant installer options should be --certs-server-cert, --certs-server-key and --certs-server-ca-cert.

5

Ohh Ok gotcha … We have to update via foreman-installer instead doing manual right ?? Let me do that right away from the snapshot recovery from the point which i started change.

6

Yes, it is always safer to do things with the installer. Doing things by hand can work, but it gets tricky to do it right at times.