Problem:
Remote Execution (Command-SSH) Fails. error while initializing command RuntimeError Failed to establish connection to remote host, exit code: 255:
Expected outcome:
Trying to use REX to run commands via SSH.
Foreman and Proxy versions:
Foreman 3.3
Foreman Proxy 3.3
Foreman and Proxy plugin versions:
|Name|Version|
|foreman-tasks|6.0.2|
|foreman_ansible|7.1.0|
|foreman_dhcp_browser|0.0.8|
|foreman_discovery|21.0.1|
|foreman_kubevirt|0.1.9|
|foreman_monitoring|2.1.0|
|foreman_openscap|5.2.2|
|foreman_puppet|4.0.1|
|foreman_remote_execution|7.1.0|
|foreman_snapshot_management|2.0.1|
|foreman_templates|9.3.0|
Distribution and version:
Other relevant data:
I have verified that ssh to the client with foreman-proxy (with keys) and passwordless sudo works correctly. Have been working on this for several days and am not able to get to the bottom of why this continues to fail. This foreman server was upgraded from prior versions of foreman.
[root@foreman01 foreman]# rpm -qa | grep fore
foreman-release-3.3.0-1.el7.noarch
tfm-rubygem-foreman-tasks-6.0.2-1.fm3_3.el7.noarch
tfm-rubygem-foreman_openscap-5.2.2-2.fm3_3.el7.noarch
ansible-collection-theforeman-foreman-3.1.0-2.el7.noarch
foreman-debug-3.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el7.noarch
tfm-rubygem-foreman_ansible-7.1.0-3.fm3_3.el7.noarch
foreman-libvirt-3.3.0-1.el7.noarch
foreman-installer-3.3.0-1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.8-1.el7.noarch
tfm-rubygem-foreman_snapshot_management-2.0.1-1.fm2_6.el7.noarch
tfm-rubygem-hammer_cli_foreman-3.3.0-1.el7.noarch
foreman-dynflow-sidekiq-3.3.0-1.el7.noarch
foreman-ovirt-3.3.0-1.el7.noarch
tfm-rubygem-foreman_templates-9.3.0-1.fm3_3.el7.noarch
tfm-rubygem-hammer_cli_foreman_puppet-0.0.6-1.fm3_3.el7.noarch
foreman-selinux-3.3.0-1.el7.noarch
tfm-rubygem-foreman_dhcp_browser-0.0.8-6.fm3_3.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.6-1.fm2_5.el7.noarch
tfm-rubygem-foreman_remote_execution-7.1.0-1.fm3_3.el7.noarch
foreman-service-3.3.0-1.el7.noarch
foreman-postgresql-3.3.0-1.el7.noarch
tfm-rubygem-foreman_puppet-4.0.1-1.fm3_3.el7.noarch
foreman-cli-3.3.0-1.el7.noarch
foreman-ec2-3.3.0-1.el7.noarch
tfm-rubygem-foreman_remote_execution-cockpit-7.1.0-1.fm3_3.el7.noarch
foreman-proxy-3.3.0-1.el7.noarch
tfm-rubygem-foreman_discovery-21.0.1-2.fm3_3.el7.noarch
foreman-vmware-3.3.0-1.el7.noarch
tfm-rubygem-foreman_ansible_core-4.1.2-1.fm2_5.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el7.noarch
foreman-3.3.0-1.el7.noarch
tfm-rubygem-foreman_kubevirt-0.1.9-4.fm3_3.el7.noarch
tfm-rubygem-foreman_monitoring-2.1.0-4.fm3_3.el7.noarch
Action:
Actions::RemoteExecution::RunHostsJob
Input:
{“job_invocation”=>{“id”=>97, “name”=>“Commands”, “description”=>“Run date”},
“job_category”=>“Commands”,
“job_invocation_id”=>97,
“proxy_batch_size”=>100,
“trigger_run_step_id”=>5,
“current_request_id”=>“3cf9d421-5fb5-4c8d-9abe-b28b19f16bd3”,
“current_timezone”=>“America/New_York”,
“current_organization_id”=>1,
“current_location_id”=>2,
“current_user_id”=>4}
Output:
{“host_count”=>1,
“remote_triggered_count”=>0,
“planned_count”=>1,
“cancelled_count”=>0,
“total_count”=>1,
“failed_count”=>1,
“pending_count”=>0,
“success_count”=>0}
Exception:
RuntimeError: A sub task failed
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_sub_plans.rb:231:in check_for_errors!' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_sub_plans.rb:137:in try_to_finish’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_polling_sub_plans.rb:19:in poll' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_polling_sub_plans.rb:11:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-7.1.0/app/lib/actions/remote_execution/run_hosts_job.rb:140:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in block (3 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/execution_wrapper.rb:91:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:94:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in execute_job' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in block (2 levels) in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in block in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in local' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in block in <module:Sidekiq>' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in stats' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in global' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in block in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in with_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in with_job_hash_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in process_one' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in watchdog' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in block in safe_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’
Action:
Actions::RemoteExecution::RunHostsJob
Input:
{“job_invocation”=>{“id”=>97, “name”=>“Commands”, “description”=>“Run date”},
“job_category”=>“Commands”,
“job_invocation_id”=>97,
“proxy_batch_size”=>100,
“trigger_run_step_id”=>5,
“current_request_id”=>“3cf9d421-5fb5-4c8d-9abe-b28b19f16bd3”,
“current_timezone”=>“America/New_York”,
“current_organization_id”=>1,
“current_location_id”=>2,
“current_user_id”=>4}
Output:
{“host_count”=>1,
“remote_triggered_count”=>0,
“planned_count”=>1,
“cancelled_count”=>0,
“total_count”=>1,
“failed_count”=>1,
“pending_count”=>0,
“success_count”=>0}