Remote execution job is pending status

[foremanfix]

Problem:
Remote execution job is pending status

Expected outcome:
It suppose to finish and needs to show in console as successfull

Foreman and Proxy versions:
foreman-proxy-3.6.1

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:
I am getting below error message while i am executing remote jobs from foreman console: (from proxy.log)

2023-10-22T07:35:06 965b5e9e [E] <OpenSSL::SSL::SSLError> SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
        /usr/share/ruby/net/protocol.rb:44:in `connect_nonblock'
        /usr/share/ruby/net/protocol.rb:44:in `ssl_socket_connect'
        /usr/share/ruby/net/http.rb:1009:in `connect'
        /usr/share/ruby/net/http.rb:943:in `do_start'
        /usr/share/ruby/net/http.rb:932:in `start'
        /usr/share/ruby/net/http.rb:1483:in `request'
        /usr/share/foreman-proxy/lib/proxy/request.rb:48:in `send_request'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/callback.rb:13:in `callback'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/callback.rb:7:in `send_to_foreman_tasks'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/callback.rb:28:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:15:in `block in run'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:15:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:69:in `execute'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:122:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
        /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:56:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
        /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-10-22T07:35:14 965b5e9e [E] <Dynflow::Action::WithSubPlans::SubtaskFailedException> A sub task failed

[aruzicka]

Hi,
your smart proxy doesn’t trust Foreman’s certificate and so it fails when it tries to report the remote execution job results to Foreman. Are you using custom certificates?

[foremanfix]

@aruzicka

trust Foreman’s certificate

Yes We are using Custom certificate and its generated via Venafi. How we can get rid of from this ?? Can you please let me know where will be the location of the certificate . Currently I am confused since we made many certification changes.

[aruzicka]

@foremanfix

Currently I am confused since we made many certification changes.

Well, that makes things somewhat hard. What is the timeline of those changes? Is the smart proxy mentioned in the first post the one co-located on the foreman server or is it an external one? Do you have katello enabled?

In general you should be able to take the certs you have from venafi, point the installer at them and the installer should take care of things. The relevant installer options should be --certs-server-cert, --certs-server-key and --certs-server-ca-cert.

[foremanfix]

Ohh Ok gotcha … We have to update via foreman-installer instead doing manual right ?? Let me do that right away from the snapshot recovery from the point which i started change.

[aruzicka]

Yes, it is always safer to do things with the installer. Doing things by hand can work, but it gets tricky to do it right at times.

[brookst]

I’m running into the exact same symptoms but don’t understand the fix here. Jobs seem to run fine in foreman-proxy, until it comes to the phase “Run Proxy::Dynflow::Callback::Action”, then it throws the SSL_connect certificate verify failed (self-signed certificate in certificate chain) error. It doesn’t show any details of the request it’s trying to make, even with debugging enabled, which makes troubleshooting difficult.

I’m running foreman-installer with these custom hiera parameters set:

    foreman::ssl: true
    foreman::servername
    foreman::foreman_url
    foreman::server_ssl_key
    foreman::server_ssl_cert
    foreman::server_ssl_chain
    foreman::websockets_ssl_key
    foreman::websockets_ssl_cert
    puppet::server_foreman_ssl_ca
    puppet::server_foreman_url
    foreman_proxy::bmc: true

These are all set to a public SSL cert for the node running Foreman, Puppet and Foreman-Proxy. This is not a self-signed cert.

I’ve seen notes talking about passing the --certs-server-cert flags to foreman-installer, but it doesn’t recognise them, I think because I don’t have Katello as part of the installation?

I don’t know what service the Dynflow callback tries to contact or how to change whatever cert it has. I’d appreciate some help getting Remote Execution to actually work.

[aruzicka]

@brookst

I don’t know what service the Dynflow callback tries to contact

It tries to contact Foreman itself. I see you don’t mention any parameters that would control what certs get used (and expected on the other end) on the smart proxy side (--foreman-proxy-ssl-ca, --foreman-proxy-ssl-cert, --foreman-proxy-ssl-key), that might be worth looking into.

[brookst]

Thanks, I think the Foreman-Proxy certs had been set elsewhere, but I have explicitly added them to the hiera for foreman-installer with the keys:

foreman::proxy_ssl_key
foreman::proxy_ssl_cert
foreman::proxy_ssl_chain

set to the same certs as Foreman itself - they both run on the same node. I reran foreman-install and everything behaves as before - I get the same error when foreman-proxy tries to call back to report a successful remote execution.

I have noticed that if I keep the job page open (foreman/job_invocations/jobid), after 10 mins of polling, the job status does resolve to success. These are the foreman-proxy logs cut down a little:

16:21:01  [D]          Step e2ba516d-3d9a-4c0a-a92b-ca8276177b61: 3   pending >>   running in phase      Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
16:21:01 f422985e [D]          Step e2ba516d-3d9a-4c0a-a92b-ca8276177b61: 3   running >> suspended in phase      Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
16:21:01  [D] start runner 2790aff4-62d4-4cd9-a616-4c5a8854cd00
16:21:01  [D] Checking if private key has passphrase: ssh-keygen -y -f /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
16:21:01  [D] close: 10.60.6.13:34806
16:21:01  [D] Private key is not protected with a passphrase
16:21:01  [D] Running: ssh -o User=root -o Port=22 -o IdentityFile=/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy [...]
16:21:02  [D] Established connection using authentication method publickey
16:21:03  [D] Running: ssh -o ControlPath=/var/tmp/2790aff4-62d4-4cd9-a616-4c5a8854cd00 -o LogLevel=quiet [...] rm -rf /var/tmp/foreman-ssh-cmd-2790aff4-62d4-4cd9-a616-4c5a8854cd00
16:21:03  [D]          Step e2ba516d-3d9a-4c0a-a92b-ca8276177b61: 3 got event #<Proxy::Dynflow::Runner::Update:0x00007fa4d397db80>
16:21:03  [D]          Step e2ba516d-3d9a-4c0a-a92b-ca8276177b61: 3 suspended >>   running in phase      Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
16:21:03 f422985e [D]          Step e2ba516d-3d9a-4c0a-a92b-ca8276177b61: 3   running >>   success in phase      Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
16:21:03  [D]          Step e2ba516d-3d9a-4c0a-a92b-ca8276177b61: 6   pending >>   running in phase      Run Proxy::Dynflow::Callback::Action
16:21:03 f422985e [E] <OpenSSL::SSL::SSLError> SSL_connect returned=1 errno=0 state=error: certificate verify failed (self-signed certificate in certificate chain)
16:21:03  [D] ExecutionPlan e2ba516d-3d9a-4c0a-a92b-ca8276177b61      running >>    paused
16:21:04  [D] Sending exit request for session root@[...]
16:21:04  [D] Closing shared connection: ssh -o ControlPath=/var/tmp/2790aff4-62d4-4cd9-a616-4c5a8854cd00 -o LogLevel=quiet [...] -O exit
16:31:01  [D] accept: 10.60.6.13:54490
16:31:01  [D] Rack::Handler::WEBrick is invoked.
16:31:01 f422985e [I] Started POST /dynflow/tasks/status
16:31:01 f422985e [D] verifying remote client 10.60.6.13 against trusted_hosts ["foreman.publichostname"]
16:31:01 f422985e [I] Finished POST /dynflow/tasks/status with 200 (1.14 ms)
16:31:01  [D] close: 10.60.6.13:54490
16:31:01  [D] accept: 10.60.6.13:54494
16:31:01  [D] Rack::Handler::WEBrick is invoked.
16:31:01 f422985e [I] Started GET /dynflow/tasks/e2ba516d-3d9a-4c0a-a92b-ca8276177b61/status
16:31:01 f422985e [D] verifying remote client 10.60.6.13 against trusted_hosts ["foreman.publichostname"]
16:31:01 f422985e [I] Finished GET /dynflow/tasks/e2ba516d-3d9a-4c0a-a92b-ca8276177b61/status with 200 (4.75 ms)
16:31:01  [D] close: 10.60.6.13:54494

To me it looks like the script on the target host is what’s trying to send the call back at 16:21. I don’t know why 10 mins passes after the callback fails before something rechecks and foreman reports the success.
It looks like the noted scripts in either the foreman/proxy or job target host /var/tmp/ directory get cleaned up, I’ll try to grab those while a job is running to see what they’re doing.

Is there anything else I could try?

[cps86]

Hello

@ aruzicka
I think there’s a bug in foreman which is causing this.

I installed foreman (NOT katello) using the regular self-signed by foreman/puppet certs. I have not provided any cert-related paramters into the foreman installer. The command line which I used to install foreman was:

foreman-installer --foreman-db-host=myexternaldatabase.infra.com --foreman-db-username=foreman --foreman-db-password=‘mydatabasepassword’ --foreman-initial-admin-password myadminpassword

So, foreman and foreman-proxy are running on the same host.
I can’t see any errors for the smart proxy in the Infrastrucutre/Smart Proxy tab.
I installed remote execution plugin and I am able to execute tasks on a remote hosts… but I have a exactly the same problem as @brookst .

I configured 2 simple tasks which simply print a message on the screen. Regardless whether it’s a ansible playbook or ssh command it always almost immediately ends with “Exit status: 0”, like following:

PLAY RECAP *********************************************************************
23:
myhost.mycompany.net : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
24:
Exit status: 0

So it was executed correctly as I can see stdout, but I need to wait around 10 minutes to resolve the job status.
In the foreman smart-proxy log there’s following:

2024-12-16T14:53:30 c6273546 [E] OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed (self-signed certificate in certificate chain)
/usr/share/ruby/net/protocol.rb:46:in connect_nonblock' /usr/share/ruby/net/protocol.rb:46:in ssl_socket_connect’
/usr/share/ruby/net/http.rb:1038:in connect' /usr/share/ruby/net/http.rb:970:in do_start’
/usr/share/ruby/net/http.rb:959:in start' /usr/share/ruby/net/http.rb:1512:in request’
/usr/share/foreman-proxy/lib/proxy/request.rb:48:in send_request' /usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/callback.rb:15:in callback’
/usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/action/batch_callback.rb:16:in run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:590:in block (3 levels) in execute_run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:29:in with_progress_calculation' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:15:in run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in pass' /usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in block in run’
/usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:51:in restore_current_request_id' /usr/share/gems/gems/smart_proxy_dynflow-0.9.3/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in pass' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in run’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/world.rb:31:in execute’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:589:in block (2 levels) in execute_run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in catch’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in block in execute_run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in block in with_error_handling’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in catch' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in with_error_handling’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:583:in execute_run' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:304:in execute’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in with_meta_calculation’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/director.rb:70:in execute’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/parallel/worker.rb:16:in block in on_message' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors.rb:18:in run_user_code’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/parallel/worker.rb:15:in on_message' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in on_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/actor.rb:122:in on_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in on_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/actor.rb:56:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in process_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in process_envelope' /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in block in on_envelope’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in block (2 levels) in schedule_execution' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in block in synchronize’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in synchronize' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in synchronize’
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in block in schedule_execution' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in call’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in work' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in block in call_job’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in run_task' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in block (3 levels) in create_worker’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in loop' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in block (2 levels) in create_worker’
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in catch' /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in block in create_worker’
/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

I tried to do the trick as @tomzellner in:

Foreman-Proxy - certificate verify failed (self signed certificate in certificate chain)

I have just installed a foreman content proxy on EL9 following the docs Installing a Smart Proxy Server 3.11 on Enterprise Linux with custom certificates. It works fine. No errors registering. Running on the main server: # foreman-proxy-certs-generate --foreman-proxy-fqdn foreman9-content.example.com \ --certs-tar /root/foreman9-content.example.com-certs.tar \ --server-cert /root/certs/foreman9-content/2024/foreman9-content_cert.cer \ --server-key /root/certs/foreman…

So I changed 64th line in the following file:

/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb

:ca_file => “/etc/puppetlabs/puppet/ssl/certs/ca.pem”

but it didn’t help.
My foreman proxy and foreman config (I HAVE NOT CHANGED ANYTHING AFTER INSTALLATION!):

Foreman:

:unattended: true
:require_ssl: true

:oauth_active: true
:oauth_map_users: false
:oauth_consumer_key: stWw3ErnxNGuUENTAiS3PKUf56rRUCzF
:oauth_consumer_secret: dYM8XeSBHwZnxhVF3TyxgN3cgCduMKCq

Websockets

:websockets_encrypt: true
:websockets_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/myservername.infra.net.pem
:websockets_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/myservername.infra.net.pem

SSL-settings

:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/myservername.infra.net.pem
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_priv_key: /etc/puppetlabs/puppet/ssl/private_keys/myservername.infra.net.pem

Foreman Proxy:

:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/myservername.infra.net.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/myservername.infra.net.pem

#:foreman_ssl_ca: ssl/certs/ca.pem
#:foreman_ssl_cert: ssl/certs/fqdn.pem
#:foreman_ssl_key: ssl/private_keys/fqdn.pem

I think this bug was introducted after 3.8 release because I didn’t see this behaviour over there.

[aruzicka]

Alright, let’s take this apart.

@cps86

foreman-installer --foreman-db-host=myexternaldatabase.infra.com --foreman-db-username=foreman --foreman-db-password=‘mydatabasepassword’ --foreman-initial-admin-password myadminpassword

This sets up a vanilla foreman without any plugins with an external database. So far so good, there doesn’t seem to be any problem, this seems to work for me just fine.

@cps86

I installed remote execution plugin

So you first ran the installer with the options listed above and then ran it again with remote-execution-specific options or how did you install it?

@cps86

In the foreman smart-proxy log there’s following:

In this case, there’s no need to paste the whole thing, it is always the same

@cps86

So I changed 64th line in the following file:

/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb

:ca_file => “/etc/puppetlabs/puppet/ssl/certs/ca.pem”

but it didn’t help.

I would be really surprised if this actually changed anything about the problem you’re having, but it was worth a try.

@cps86

I think this bug was introducted after 3.8 release because I didn’t see this behaviour over there.

While I can’t refute this, running the following sequence on a fresh alma 9.5 box works for me just fine

foreman-installer --foreman-db-host=extdb.example.com --foreman-db-username=foreman --foreman-db-password=password --foreman-initial-admin-password changeme
foreman-installer --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-script

After doing this, I can register a host and run a job against it and it finishes in a couple of seconds. While there might be a bug somewhere, I had to luck in triggering it.

[cps86]

Hey @aruzicka

Thanks for replying. Okay, you’re right. I re-deploy everything over again.
It seems the problem is with my second foreman smart proxy server.
What I wanted to achieve is some kind of HA-setup which 2 foreman servers (and smart proxies) are using the same foreman database. What I did was:

1. Installed foreman + all the plugins on SERVER1. I can successfully execute tasks from that server, they end up immediately. I do not get any SSL errors.
2. I deployed second instance of foreman on SERVER2. For this purpose, on SERVER1 I executed the following:

puppetserver ca generate --certname server2.net

Successfully saved private key for server2.net to /etc/puppetlabs/puppet/ssl/private_keys/server2.net.pem|
||Successfully saved public key for server2.net to /etc/puppetlabs/puppet/ssl/public_keys/server2.net.pem|
||Successfully submitted certificate request for server2.net|
||Error:|
|| Signed certificate server2.net could not be found on the CA|
||Successfully signed certificate request for server2.net|
||Successfully saved certificate for server2.net to /etc/puppetlabs/puppet/ssl/certs/server2.net.pem|

3. From SERVER1 I copied the following files to SERVER2:

CA:
/etc/puppetlabs/puppet/ssl/certs/ca.pem
Private key:
/etc/puppetlabs/puppet/ssl/private_keys/server2.net.pem
Cert:
/etc/puppetlabs/puppet/ssl/certs/server2.net.pem

4. On SERVER2 I copied the ca cert to the ca-trust folder:

cp ca_crt.pem /etc/pki/ca-trust/source/anchors/
update-ca-trust

5. Then installed foreman on SERVER2 by executing:

foreman-installer --foreman-db-host=mydatabaseserver.net --foreman-db-username=foreman --foreman-db-password='mydbpassword' --foreman-initial-admin-password myadminpassword --foreman-websockets-ssl-cert=/etc/certs/server2.net.pem --foreman-websockets-ssl-key=/etc/certs/server2.net.key --foreman-server-ssl-ca=/etc/certs/ca_crt.pem --foreman-server-ssl-cert=/etc/certs/server2.net.pem --foreman-server-ssl-key=/etc/certs/server2.net.key --foreman-proxy-ssl-ca=/etc/certs/ca_crt.pem --foreman-proxy-ssl-cert=/etc/certs/server2.net.pem --foreman-proxy-ssl-key=/etc/certs/server2.net.key --foreman-client-ssl-ca=/etc/certs/ca_crt.pem --foreman-client-ssl-cert=/etc/certs/server2.net.pem --foreman-client-ssl-key=/etc/certs/server2.net.key --foreman-proxy-trusted-hosts=server2.net --foreman-proxy-trusted-hosts=server1.net --puppet-runmode none --puppet-server false --foreman-proxy-puppet false --foreman-proxy-puppetca false --foreman-server-ssl-chain=/etc/certs/ca_crt.pem --foreman-server-ssl-crl=/etc/certs/ca_crt.pem

Later on I also tried to provide following arguments with the same values as they are set on server1:
--foreman-proxy-oauth-consumer-key --foreman-proxy-oauth-consumer-secret
But it didn’t make any difference.

Foreman on Server2 was successfully installed without any errors. Same with plugins.
But for some reason while executing tasks from Server2 I am getting:

SSL_read: tlsv1 alert unknown ca

Why is that? Can you please check if my ssl-related arguments provided as the installation parameters are correct?

Please help.

Kind regards,
Luke

[aruzicka]

@cps86

What I wanted to achieve is some kind of HA-setup which 2 foreman servers (and smart proxies) are using the same foreman database.

Oof, so there’s foreman1 and smartproxy1 on one machine and foreman2 and smartproxy2 on another machine, foreman1 and foreman2 talk to the same database, correct? Foreman1 and foreman2 then form a single “logical foreman”, but they don’t have a common name so this logical foreman is essentially multihomed?

Now I’m just guessing, but that would probably also mean that when you’re executing a remote execution job, the instance (foreman1 or foreman2) that is processing the job can send it to any of the smart proxies and the smartproxy then tries to call back to its foreman instance no matter from which instance the original request came so it can “cross” (foreman1 → smartproxy2 → foreman2).

@cps86

But for some reason while executing tasks from Server2 I am getting:

SSL_read: tlsv1 alert unknown ca

Could you go through the logs and trace the whole execution? Did it go foreman2 → smartproxy2 → foreman2 or did it go some other way?

@cps86

Why is that?

Honestly no idea. This doesn’t seem to be an exactly common approach