Remote Execution Plugin Failing, RuntimeError - Could not use any proxy

Problem:
Hi All,
I have the same problem like other users here, but did not find a solution
Fresh installed Foreman 3.1
with enabled plugin-remote-execution, and proxy-plugin-remote-execution-ssh
Under Smart Proxy I see my smart proxy
I have Location, Organisation Subnet configured.
all to use the proxy on my foreman server.
foreman proxy service is also Answering on port 9090.
ssh key copyied to client root.

always get

Failed to initialize: RuntimeError - Could not use any proxy. Consider configuring remote_execution_global_proxy, remote_execution_fallback_proxy in settings

the two settings are both set to yes.

what am I missing.

thanks for any help.

Thomas

Expected outcome:
remote execution working

Foreman and Proxy versions:
latest Foreman 3.1
yesterday downloaded and installed

Foreman and Proxy plugin versions:
how can I get this?
latest, yesterday downloaded installed
Distribution and version:
Foreman Server RHEL 7.9
Targeted Client Ubuntu 20.04
Other relevant data:
could post logs here if you can tell me where to find.
/var/logs/foreman/

Having /var/log/foreman/production.log and /var/log/foreman-proxy/proxy.log from around that time would be helpful

below you can find the production.log
in the proxy.log there is no log from that time.
just the “smart proxy has launched …”

2022-02-25T13:04:43 [I|app|a1b46ad3] Started GET "/notification_recipients" for 10.10.1.187 at 2022-02-25 13:04:43 +0100
2022-02-25T13:04:43 [I|app|a1b46ad3] Processing by NotificationRecipientsController#index as JSON
2022-02-25T13:04:43 [I|app|a1b46ad3] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 0.8ms | Allocations: 1789)
2022-02-25T13:04:43 [I|app|65eaf588] Started POST "/job_invocations?feature=ansible_enable_web_console&host_ids%5B%5D=4" for 10.10.1.187 at 2022-02-25 13:04:43 +0100
2022-02-25T13:04:43 [I|app|65eaf588] Processing by JobInvocationsController#create as HTML
2022-02-25T13:04:43 [I|app|65eaf588]   Parameters: {"authenticity_token"=>"cGCVN7Pv4jkn2RkxNxNiTP6dLnpHmO67wD0RZndrqjxUpEindLI0aPKppgX7iGcIXzvYIUyZ1wOtUmCdEW8UxQ==", "feature"=>"ansible_enable_web_console", "host_ids"=>["4"]}
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on job_category Ansible Services
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on description Service Action - Enable Web Console
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on concurrency_level
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on time_span
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on execution_timeout_interval
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on password [redacted]
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on key_passphrase
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on remote_execution_feature_id 15
2022-02-25T13:04:44 [I|aud|65eaf588] JobInvocation (13) create event on effective_user_password
2022-02-25T13:04:44 [I|bac|65eaf588] Task {label: , id: 22557970-2216-4e2e-93ad-846a43b50778, execution_plan_id: 52dd2f6d-bf62-48f5-ac7f-f6227dc387c0} state changed: pending
2022-02-25T13:04:44 [I|bac|65eaf588] Task {label: Actions::RemoteExecution::RunHostsJob, id: 22557970-2216-4e2e-93ad-846a43b50778, execution_plan_id: 52dd2f6d-bf62-48f5-ac7f-f6227dc387c0} state changed: planning
2022-02-25T13:04:44 [I|bac|65eaf588] Task {label: Actions::RemoteExecution::RunHostsJob, id: 22557970-2216-4e2e-93ad-846a43b50778, execution_plan_id: 52dd2f6d-bf62-48f5-ac7f-f6227dc387c0} state changed: planned
2022-02-25T13:04:44 [I|app|65eaf588] Redirected to https://test-foreman.nukular.local/job_invocations/13
2022-02-25T13:04:44 [I|app|65eaf588] Completed 302 Found in 422ms (ActiveRecord: 65.1ms | Allocations: 84077)
2022-02-25T13:04:44 [I|app|0bbc4ee5] Started GET "/job_invocations/13" for 10.10.1.187 at 2022-02-25 13:04:44 +0100
2022-02-25T13:04:44 [I|app|0bbc4ee5] Processing by JobInvocationsController#show as HTML
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Parameters: {"id"=>"13"}
2022-02-25T13:04:44 [I|bac|65eaf588] Task {label: Actions::RemoteExecution::RunHostsJob, id: 22557970-2216-4e2e-93ad-846a43b50778, execution_plan_id: 52dd2f6d-bf62-48f5-ac7f-f6227dc387c0} state changed: running
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.html.erb within layouts/application
2022-02-25T13:04:44 [I|bac|65eaf588] Task {label: , id: 7767af76-674e-4175-936e-deb9910d03bd, execution_plan_id: 3be6d79c-e3cd-44b1-ab05-afdc38c3fe1a} state changed: pending
2022-02-25T13:04:44 [I|bac|65eaf588] Task {label: Actions::RemoteExecution::RunHostJob, id: 7767af76-674e-4175-936e-deb9910d03bd, execution_plan_id: 3be6d79c-e3cd-44b1-ab05-afdc38c3fe1a} state changed: planning
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/_card_results.html.erb (Duration: 0.6ms | Allocations: 328)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/_card_schedule.html.erb (Duration: 5.1ms | Allocations: 1076)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/_card_target_hosts.html.erb (Duration: 4.4ms | Allocations: 1554)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/_tab_hosts.html.erb (Duration: 0.8ms | Allocations: 269)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/_tab_overview.html.erb (Duration: 48.3ms | Allocations: 12343)
2022-02-25T13:04:44 [E|bac|65eaf588] Could not use any proxy for the Ansible job. Consider configuring remote_execution_global_proxy, remote_execution_fallback_proxy in settings (RuntimeError)
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_host_job.rb:227:in `determine_proxy!'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_host_job.rb:44:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:525:in `block (3 levels) in execute_plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:15:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:34:in `with_current_request_id'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:41:in `with_current_user'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:524:in `block (2 levels) in execute_plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:387:in `switch_flow'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:428:in `concurrence'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:523:in `block in execute_plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:522:in `execute_plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:289:in `block (2 levels) in plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:387:in `switch_flow'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:377:in `with_planning_scope'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:288:in `block in plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:287:in `plan'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:219:in `block (2 levels) in plan_with_options'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator.rb:326:in `acquire'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:217:in `block in plan_with_options'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:216:in `tap'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:216:in `plan_with_options'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:84:in `block in trigger'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:178:in `trigger'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:84:in `trigger'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_hosts_job.rb:51:in `block in create_sub_plans'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_hosts_job.rb:46:in `map'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_hosts_job.rb:46:in `create_sub_plans'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:41:in `spawn_plans'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_bulk_sub_plans.rb:71:in `spawn_plans'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_hosts_job.rb:56:in `spawn_plans'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:37:in `initiate'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_bulk_sub_plans.rb:36:in `initiate'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_polling_sub_plans.rb:24:in `initiate'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_hosts_job.rb:110:in `initiate'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:17:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:12:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_bulk_sub_plans.rb:24:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_polling_sub_plans.rb:13:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_hosts_job.rb:140:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 65eaf588 | [ sidekiq ]
 65eaf588 | [ concurrent-ruby ]
2022-02-25T13:04:44 [I|bac|65eaf588] Task {label: Actions::RemoteExecution::RunHostJob, id: 7767af76-674e-4175-936e-deb9910d03bd, execution_plan_id: 3be6d79c-e3cd-44b1-ab05-afdc38c3fe1a} state changed: stopped  result: error
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/_user_input.html.erb (Duration: 16.1ms | Allocations: 5320)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/_tab_preview_templates.html.erb (Duration: 67.3ms | Allocations: 8596)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.html.erb within layouts/application (Duration: 168.7ms | Allocations: 33925)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered layouts/_application_content.html.erb (Duration: 2.5ms | Allocations: 5333)
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendering layouts/base.html.erb
2022-02-25T13:04:44 [I|app|0bbc4ee5]   Rendered layouts/base.html.erb (Duration: 19.8ms | Allocations: 23093)
2022-02-25T13:04:44 [I|app|0bbc4ee5] Completed 200 OK in 245ms (Views: 184.6ms | ActiveRecord: 18.1ms | Allocations: 77318)
2022-02-25T13:04:44 [I|app|17d74a52] Started GET "/javascripts/charts.js" for 10.10.1.187 at 2022-02-25 13:04:44 +0100
2022-02-25T13:04:44 [F|app|17d74a52]
 17d74a52 | ActionController::RoutingError (No route matches [GET] "/javascripts/charts.js"):
 17d74a52 |
 17d74a52 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 17d74a52 | katello (4.3.0) lib/katello/prevent_json_parsing.rb:12:in `call'
2022-02-25T13:04:44 [I|app|ba300a8a] Started GET "/notification_recipients" for 10.10.1.187 at 2022-02-25 13:04:44 +0100
2022-02-25T13:04:44 [I|app|ba300a8a] Processing by NotificationRecipientsController#index as JSON
2022-02-25T13:04:44 [I|app|ba300a8a] Completed 200 OK in 6ms (Views: 0.1ms | ActiveRecord: 0.8ms | Allocations: 1805)
2022-02-25T13:04:44 [I|app|538fd920] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:44 +0100
2022-02-25T13:04:44 [I|app|538fd920] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:44 [I|app|538fd920]   Parameters: {"id"=>"13"}
2022-02-25T13:04:45 [I|app|538fd920] Completed 200 OK in 22ms (Views: 0.2ms | ActiveRecord: 4.5ms | Allocations: 7450)
2022-02-25T13:04:45 [I|app|ccfb1364] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:45 +0100
2022-02-25T13:04:45 [I|app|ccfb1364] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:45 [I|app|ccfb1364]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:45 [I|app|ccfb1364]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:45 [I|app|ccfb1364]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 10.4ms | Allocations: 6292)
2022-02-25T13:04:45 [I|app|ccfb1364] Completed 200 OK in 32ms (Views: 10.4ms | ActiveRecord: 4.4ms | Allocations: 16030)
2022-02-25T13:04:45 [I|app|260d558d] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:45 +0100
2022-02-25T13:04:45 [I|app|260d558d] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:45 [I|app|260d558d]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:45 [I|app|260d558d]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:45 [I|app|260d558d]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 8.5ms | Allocations: 6272)
2022-02-25T13:04:45 [I|app|260d558d] Completed 200 OK in 34ms (Views: 8.8ms | ActiveRecord: 5.0ms | Allocations: 16004)
2022-02-25T13:04:46 [I|app|f679054a] Started GET "/job_invocations/13?_=1645790684871" for 10.10.1.187 at 2022-02-25 13:04:46 +0100
2022-02-25T13:04:46 [I|app|f679054a] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:46 [I|app|f679054a]   Parameters: {"_"=>"1645790684871", "id"=>"13"}
2022-02-25T13:04:46 [I|app|f679054a]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:46 [I|app|f679054a]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 25.4ms | Allocations: 12610)
2022-02-25T13:04:46 [I|app|f679054a] Completed 200 OK in 46ms (Views: 23.0ms | ActiveRecord: 6.4ms | Allocations: 20612)
2022-02-25T13:04:46 [I|app|6666ffe8] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:46 +0100
2022-02-25T13:04:46 [I|app|6666ffe8] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:46 [I|app|6666ffe8]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:46 [I|app|6666ffe8]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:46 [I|app|6666ffe8]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 5.3ms | Allocations: 6272)
2022-02-25T13:04:46 [I|app|6666ffe8] Completed 200 OK in 24ms (Views: 5.6ms | ActiveRecord: 3.0ms | Allocations: 16010)
2022-02-25T13:04:46 [I|app|5e29fb10] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:46 +0100
2022-02-25T13:04:46 [I|app|5e29fb10] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:46 [I|app|5e29fb10]   Parameters: {"id"=>"13"}
2022-02-25T13:04:46 [I|app|5e29fb10] Completed 200 OK in 14ms (Views: 0.2ms | ActiveRecord: 1.8ms | Allocations: 7639)
2022-02-25T13:04:47 [I|app|b7a4ae0c] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684872" for 10.10.1.187 at 2022-02-25 13:04:47 +0100
2022-02-25T13:04:47 [I|app|b7a4ae0c] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:47 [I|app|b7a4ae0c]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684872", "id"=>"13"}
2022-02-25T13:04:47 [I|app|b7a4ae0c]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:47 [I|app|b7a4ae0c]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 15.7ms | Allocations: 7867)
2022-02-25T13:04:47 [I|app|b7a4ae0c] Completed 200 OK in 39ms (Views: 15.8ms | ActiveRecord: 10.0ms | Allocations: 19118)
2022-02-25T13:04:47 [I|app|588e9e73] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:47 +0100
2022-02-25T13:04:47 [I|app|588e9e73] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:47 [I|app|588e9e73]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:47 [I|app|588e9e73]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:47 [I|app|588e9e73]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 8.8ms | Allocations: 6345)
2022-02-25T13:04:47 [I|app|588e9e73] Completed 200 OK in 31ms (Views: 8.6ms | ActiveRecord: 7.9ms | Allocations: 16086)
2022-02-25T13:04:47 [I|app|14498729] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:47 +0100
2022-02-25T13:04:47 [I|app|14498729] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:47 [I|app|14498729]   Parameters: {"id"=>"13"}
2022-02-25T13:04:47 [I|app|14498729] Completed 200 OK in 17ms (Views: 0.4ms | ActiveRecord: 2.7ms | Allocations: 7547)
2022-02-25T13:04:48 [I|app|b81880bc] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684873" for 10.10.1.187 at 2022-02-25 13:04:48 +0100
2022-02-25T13:04:48 [I|app|b81880bc] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:48 [I|app|b81880bc]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684873", "id"=>"13"}
2022-02-25T13:04:48 [I|app|b81880bc]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:48 [I|app|b81880bc]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 18.8ms | Allocations: 12560)
2022-02-25T13:04:48 [I|app|b81880bc] Completed 200 OK in 32ms (Views: 17.2ms | ActiveRecord: 4.2ms | Allocations: 20559)
2022-02-25T13:04:48 [I|app|a0c9d9c0] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:48 +0100
2022-02-25T13:04:48 [I|app|a0c9d9c0] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:48 [I|app|a0c9d9c0]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:48 [I|app|a0c9d9c0]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:48 [I|app|a0c9d9c0]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 7.6ms | Allocations: 6333)
2022-02-25T13:04:48 [I|app|a0c9d9c0] Completed 200 OK in 26ms (Views: 7.7ms | ActiveRecord: 2.7ms | Allocations: 16072)
2022-02-25T13:04:48 [I|app|94516980] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:48 +0100
2022-02-25T13:04:48 [I|app|94516980] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:48 [I|app|94516980]   Parameters: {"id"=>"13"}
2022-02-25T13:04:48 [I|app|94516980] Completed 200 OK in 12ms (Views: 0.6ms | ActiveRecord: 1.4ms | Allocations: 7544)
2022-02-25T13:04:49 [I|app|63409c0c] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684874" for 10.10.1.187 at 2022-02-25 13:04:49 +0100
2022-02-25T13:04:49 [I|app|63409c0c] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:49 [I|app|63409c0c]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684874", "id"=>"13"}
2022-02-25T13:04:49 [I|app|63409c0c]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:49 [I|app|63409c0c]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 21.3ms | Allocations: 12945)
2022-02-25T13:04:49 [I|app|63409c0c] Completed 200 OK in 33ms (Views: 19.4ms | ActiveRecord: 4.0ms | Allocations: 20944)
2022-02-25T13:04:49 [I|app|e6164d4d] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:49 +0100
2022-02-25T13:04:49 [I|app|e6164d4d] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:49 [I|app|e6164d4d]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:49 [I|app|e6164d4d]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:49 [I|app|e6164d4d]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 5.5ms | Allocations: 6333)
2022-02-25T13:04:49 [I|app|e6164d4d] Completed 200 OK in 29ms (Views: 8.5ms | ActiveRecord: 6.6ms | Allocations: 16066)
2022-02-25T13:04:50 [I|app|4cb68eb3] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:50 +0100
2022-02-25T13:04:50 [I|app|4cb68eb3] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:50 [I|app|4cb68eb3]   Parameters: {"id"=>"13"}
2022-02-25T13:04:50 [I|app|4cb68eb3] Completed 200 OK in 16ms (Views: 0.3ms | ActiveRecord: 3.1ms | Allocations: 7448)
2022-02-25T13:04:50 [I|app|0e532360] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684875" for 10.10.1.187 at 2022-02-25 13:04:50 +0100
2022-02-25T13:04:50 [I|app|0e532360] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:50 [I|app|0e532360]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684875", "id"=>"13"}
2022-02-25T13:04:50 [I|app|0e532360]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:50 [I|app|0e532360]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 16.2ms | Allocations: 12560)
2022-02-25T13:04:50 [I|app|0e532360] Completed 200 OK in 28ms (Views: 15.4ms | ActiveRecord: 3.2ms | Allocations: 20559)
2022-02-25T13:04:50 [I|app|84cc2f91] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:50 +0100
2022-02-25T13:04:50 [I|app|84cc2f91] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:50 [I|app|84cc2f91]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:50 [I|app|84cc2f91]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:50 [I|app|84cc2f91]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 5.9ms | Allocations: 6237)
2022-02-25T13:04:50 [I|app|84cc2f91] Completed 200 OK in 21ms (Views: 6.1ms | ActiveRecord: 2.2ms | Allocations: 15969)
2022-02-25T13:04:51 [I|app|ee376246] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:51 +0100
2022-02-25T13:04:51 [I|app|ee376246] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:51 [I|app|ee376246]   Parameters: {"id"=>"13"}
2022-02-25T13:04:51 [I|app|ee376246] Completed 200 OK in 12ms (Views: 0.2ms | ActiveRecord: 1.5ms | Allocations: 7448)
2022-02-25T13:04:51 [I|app|f2e818f0] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684876" for 10.10.1.187 at 2022-02-25 13:04:51 +0100
2022-02-25T13:04:51 [I|app|f2e818f0] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:51 [I|app|f2e818f0]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684876", "id"=>"13"}
2022-02-25T13:04:51 [I|app|f2e818f0]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:51 [I|app|f2e818f0]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 21.3ms | Allocations: 12560)
2022-02-25T13:04:51 [I|app|f2e818f0] Completed 200 OK in 41ms (Views: 18.9ms | ActiveRecord: 5.3ms | Allocations: 20559)
2022-02-25T13:04:51 [I|app|c9e3d726] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:51 +0100
2022-02-25T13:04:51 [I|app|c9e3d726] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:51 [I|app|c9e3d726]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:51 [I|app|c9e3d726]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:51 [I|app|c9e3d726]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 6.0ms | Allocations: 6237)
2022-02-25T13:04:51 [I|app|c9e3d726] Completed 200 OK in 28ms (Views: 6.4ms | ActiveRecord: 3.2ms | Allocations: 15969)
2022-02-25T13:04:52 [I|app|631cd915] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:52 +0100
2022-02-25T13:04:52 [I|app|631cd915] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:52 [I|app|631cd915]   Parameters: {"id"=>"13"}
2022-02-25T13:04:52 [I|app|631cd915] Completed 200 OK in 13ms (Views: 0.2ms | ActiveRecord: 1.5ms | Allocations: 7576)
2022-02-25T13:04:52 [I|app|90d32748] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684877" for 10.10.1.187 at 2022-02-25 13:04:52 +0100
2022-02-25T13:04:52 [I|app|90d32748] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:52 [I|app|90d32748]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684877", "id"=>"13"}
2022-02-25T13:04:52 [I|app|90d32748]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:52 [I|app|90d32748]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 17.5ms | Allocations: 12944)
2022-02-25T13:04:52 [I|app|90d32748] Completed 200 OK in 33ms (Views: 16.0ms | ActiveRecord: 4.0ms | Allocations: 20943)
2022-02-25T13:04:52 [I|app|6727c92d] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:52 +0100
2022-02-25T13:04:52 [I|app|6727c92d] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:52 [I|app|6727c92d]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:52 [I|app|6727c92d]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:52 [I|app|6727c92d]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 4.5ms | Allocations: 6237)
2022-02-25T13:04:52 [I|app|6727c92d] Completed 200 OK in 21ms (Views: 4.8ms | ActiveRecord: 2.8ms | Allocations: 15969)
2022-02-25T13:04:53 [I|app|7450f015] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:53 +0100
2022-02-25T13:04:53 [I|app|7450f015] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:53 [I|app|7450f015]   Parameters: {"id"=>"13"}
2022-02-25T13:04:53 [I|app|7450f015] Completed 200 OK in 13ms (Views: 0.2ms | ActiveRecord: 1.8ms | Allocations: 7577)
2022-02-25T13:04:53 [I|app|6014cb9a] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684878" for 10.10.1.187 at 2022-02-25 13:04:53 +0100
2022-02-25T13:04:53 [I|app|6014cb9a] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:53 [I|app|6014cb9a]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684878", "id"=>"13"}
2022-02-25T13:04:53 [I|app|6014cb9a]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:53 [I|app|6014cb9a]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 22.6ms | Allocations: 12945)
2022-02-25T13:04:53 [I|app|6014cb9a] Completed 200 OK in 37ms (Views: 20.2ms | ActiveRecord: 5.1ms | Allocations: 20944)
2022-02-25T13:04:53 [I|app|b586d028] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:53 +0100
2022-02-25T13:04:53 [I|app|b586d028] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:53 [I|app|b586d028]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:53 [I|app|b586d028]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:53 [I|app|b586d028]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 4.4ms | Allocations: 6237)
2022-02-25T13:04:53 [I|app|b586d028] Completed 200 OK in 21ms (Views: 4.8ms | ActiveRecord: 2.8ms | Allocations: 15969)
2022-02-25T13:04:54 [I|app|1d43cbc9] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:54 +0100
2022-02-25T13:04:54 [I|app|1d43cbc9] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:54 [I|app|1d43cbc9]   Parameters: {"id"=>"13"}
2022-02-25T13:04:54 [I|app|1d43cbc9] Completed 200 OK in 12ms (Views: 0.2ms | ActiveRecord: 1.8ms | Allocations: 7448)
2022-02-25T13:04:54 [E|bac|65eaf588] A sub task failed (RuntimeError)
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:231:in `check_for_errors!'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:137:in `try_to_finish'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_polling_sub_plans.rb:19:in `poll'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_polling_sub_plans.rb:11:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/lib/actions/remote_execution/run_hosts_job.rb:140:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:94:in `execute'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 65eaf588 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 65eaf588 | [ sidekiq ]
 65eaf588 | [ concurrent-ruby ]
2022-02-25T13:04:54 [I|bac|65eaf588] Task {label: Actions::RemoteExecution::RunHostsJob, id: 22557970-2216-4e2e-93ad-846a43b50778, execution_plan_id: 52dd2f6d-bf62-48f5-ac7f-f6227dc387c0} state changed: stopped  result: warning
2022-02-25T13:04:54 [I|app|abb8ccc3] Started GET "/job_invocations/13?hosts_needs_refresh=&_=1645790684879" for 10.10.1.187 at 2022-02-25 13:04:54 +0100
2022-02-25T13:04:54 [I|app|abb8ccc3] Processing by JobInvocationsController#show as JS
2022-02-25T13:04:54 [I|app|abb8ccc3]   Parameters: {"hosts_needs_refresh"=>"", "_"=>"1645790684879", "id"=>"13"}
2022-02-25T13:04:54 [I|app|abb8ccc3]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb
2022-02-25T13:04:54 [I|app|abb8ccc3]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.js.erb (Duration: 23.7ms | Allocations: 13538)
2022-02-25T13:04:54 [I|app|abb8ccc3] Completed 200 OK in 52ms (Views: 16.3ms | ActiveRecord: 13.2ms | Allocations: 21952)
2022-02-25T13:04:54 [I|app|4200e656] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:54 +0100
2022-02-25T13:04:54 [I|app|4200e656] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:54 [I|app|4200e656]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:54 [I|app|4200e656]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:54 [I|app|4200e656]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 5.9ms | Allocations: 6238)
2022-02-25T13:04:54 [I|app|4200e656] Completed 200 OK in 24ms (Views: 6.0ms | ActiveRecord: 3.1ms | Allocations: 16228)
2022-02-25T13:04:54 [I|app|fd5a7c5d] Started GET "/job_invocations/13?page=1&per_page=20" for 10.10.1.187 at 2022-02-25 13:04:54 +0100
2022-02-25T13:04:54 [I|app|fd5a7c5d] Processing by JobInvocationsController#show as JSON
2022-02-25T13:04:54 [I|app|fd5a7c5d]   Parameters: {"page"=>"1", "per_page"=>"20", "id"=>"13"}
2022-02-25T13:04:54 [I|app|fd5a7c5d]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb
2022-02-25T13:04:54 [I|app|fd5a7c5d]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/views/job_invocations/show.json.erb (Duration: 5.0ms | Allocations: 6237)
2022-02-25T13:04:54 [I|app|fd5a7c5d] Completed 200 OK in 21ms (Views: 5.9ms | ActiveRecord: 2.2ms | Allocations: 16227)
2022-02-25T13:04:54 [I|app|14f0e719] Started GET "/notification_recipients" for 10.10.1.187 at 2022-02-25 13:04:54 +0100
2022-02-25T13:04:54 [I|app|14f0e719] Processing by NotificationRecipientsController#index as JSON
2022-02-25T13:04:55 [I|app|14f0e719] Completed 200 OK in 15ms (Views: 0.1ms | ActiveRecord: 2.6ms | Allocations: 7133)
2022-02-25T13:04:55 [I|app|4b976174] Started GET "/job_invocations/chart?id=13" for 10.10.1.187 at 2022-02-25 13:04:55 +0100
2022-02-25T13:04:55 [I|app|4b976174] Processing by JobInvocationsController#chart as JSON
2022-02-25T13:04:55 [I|app|4b976174]   Parameters: {"id"=>"13"}
2022-02-25T13:04:55 [I|app|4b976174] Completed 200 OK in 12ms (Views: 0.2ms | ActiveRecord: 1.5ms | Allocations: 7728)
2022-02-25T13:05:04 [I|app|6f64bb68] Started GET "/notification_recipients" for 10.10.1.187 at 2022-02-25 13:05:04 +0100
2022-02-25T13:05:04 [I|app|6f64bb68] Processing by NotificationRecipientsController#index as JSON
2022-02-25T13:05:04 [I|app|6f64bb68] Completed 200 OK in 4ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 1789)
2022-02-25T13:05:14 [I|app|2994d15c] Started GET "/notification_recipients" for 10.10.1.187 at 2022-02-25 13:05:14 +0100
2022-02-25T13:05:14 [I|app|2994d15c] Processing by NotificationRecipientsController#index as JSON
2022-02-25T13:05:14 [I|app|2994d15c] Completed 200 OK in 4ms (Views: 0.1ms | ActiveRecord: 0.5ms | Allocations: 1789)
2022-02-25T13:05:24 [I|app|7c13c1ba] Started GET "/notification_recipients" for 10.10.1.187 at 2022-02-25 13:05:24 +0100
2022-02-25T13:05:24 [I|app|7c13c1ba] Processing by NotificationRecipientsController#index as JSON
2022-02-25T13:05:24 [I|app|7c13c1ba] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.9ms | Allocations: 1789)

ok, some news:
the job I tried, was always the “Enable web console” job from Hosts-View.
Did not even tried execution other jobs.
But now I tried a simple “command” Job and “update deb package” from content-host view.
both working. and here in proxy.log I see the execution.

for “enable web console” still the same error.
strange

Aha! Enable web console uses ansible, which you to need to install explicitly. Try rerunning the installer with --enable-foreman-proxy-plugin-ansible

o yeah that fixed it.
was just confused by the error message.
Thanks