Remote execution failure and to idea why :/

tybreizh29 · May 3, 2018, 2:11pm

Problem:
I want to execute a “uname” via ssh on a node, but something is wrong and i did not find out what’s wrong.
/var/log/foreman/production.log

2018-05-03 15:53:30 a73f6aa5 [app] [I] Started POST “/job_invocations” for 10.22.42.88 at 2018-05-03 15:53:30 +0200
2018-05-03 15:53:30 a73f6aa5 [app] [I] Processing by JobInvocationsController#create as HTML
2018-05-03 15:53:30 a73f6aa5 [app] [I] Parameters: {“utf8”=>“✓”, “authenticity_token”=>“lu6GjrVO52SMFHbFdhXF2xtv/QW8ZMAZiqsd13Wfyl8rSrjJ9pMBACMmq1ZCDCnFgM6PR5+XrPUGzbr+wcuhiw==”, “job_invocation”=>{“job_category”=>“Commands”, “providers”=>{“SSH”=>{“job_template_id”=>“99”, “job_templates”=>{“99”=>{“input_values”=>“[FILTERED]”, “effective_user”=>“”, “execution_timeout_interval”=>“”}}}}, “description”=>“Run uname”, “description_override”=>“Run %{command}”, “description_format”=>“Run %{command}”, “concurrency_level”=>“”, “time_span”=>“”}, “targeting”=>{“bookmark_id”=>“”, “search_query”=>“name ^ (scvmpr15.fr.xxx.priv)”, “targeting_type”=>“static_query”}, “triggering”=>{“mode”=>“immediate”, “start_at_raw”=>“2018-05-03 13:53”, “start_before_raw”=>“”, “input_type”=>“daily”, “cronline”=>“”, “days”=>“”, “days_of_week”=>{“1”=>“0”, “2”=>“0”, “3”=>“0”, “4”=>“0”, “5”=>“0”, “6”=>“0”, “7”=>“0”}, “time”=>{“time(1i)”=>“2018”, “time(2i)”=>“5”, “time(3i)”=>“3”, “time(4i)”=>“13”, “time(5i)”=>“53”}, “max_iteration”=>“”, “end_time_limited”=>“false”, “end_time”=>{“end_time(3i)”=>“3”, “end_time(2i)”=>“5”, “end_time(1i)”=>“2018”, “end_time(4i)”=>“13”, “end_time(5i)”=>“53”}}, “commit”=>“Valider”}
2018-05-03 15:53:30 a73f6aa5 [app] [I] Current user: admin (administrator)
2018-05-03 15:53:30 a73f6aa5 [app] [I] Current user: admin (administrator)
2018-05-03 15:53:30 a73f6aa5 [app] [I] Current user: admin (administrator)
2018-05-03 15:53:30 a73f6aa5 [app] [I] Redirected to https://scvmpr15.fr.xxx.priv/job_invocations/30
2018-05-03 15:53:30 a73f6aa5 [app] [I] Completed 302 Found in 527ms (ActiveRecord: 196.5ms)
2018-05-03 15:53:30 206d9d55 [app] [I] Started GET “/job_invocations/30” for 10.22.42.88 at 2018-05-03 15:53:30 +0200
2018-05-03 15:53:30 206d9d55 [app] [I] Processing by JobInvocationsController#show as HTML
2018-05-03 15:53:30 206d9d55 [app] [I] Parameters: {“id”=>“30”}
2018-05-03 15:53:30 206d9d55 [app] [I] Current user: admin (administrator)
…
2018-05-03 15:53:30 206d9d55 [app] [I] Completed 200 OK in 374ms (Views: 211.4ms | ActiveRecord: 149.1ms)
2018-05-03 15:53:32 ff200dc9 [app] [I] Started GET “/notification_recipients” for 10.22.42.88 at 2018-05-03 15:53:32 +0200
2018-05-03 15:53:32 ff200dc9 [app] [I] Processing by NotificationRecipientsController#index as JSON
2018-05-03 15:53:32 ff200dc9 [app] [I] Current user: admin (administrator)
2018-05-03 15:53:32 ff200dc9 [app] [I] Completed 200 OK in 6ms (Views: 0.2ms | ActiveRecord: 1.8ms)
2018-05-03 15:53:32 [app] [W] Could not fetch task counts from scvmpr15.fr.xxx.priv, skipped.
| RestClient::NotFound: 404 Not Found
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in exception_with_response' | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in return!’
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:809:in process_result' | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in block in transmit’
| /opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in start' | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in transmit’
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in get' | /usr/share/foreman/lib/proxy_api/resource.rb:62:in block in get’
| /usr/share/foreman/lib/proxy_api/resource.rb:95:in with_logger' | /usr/share/foreman/lib/proxy_api/resource.rb:59:in get’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/proxy_api/foreman_dynflow/dynflow_proxy.rb:35:in tasks_count' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:48:in block in get_counts’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:45:in each' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:45:in each_with_object’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:45:in get_counts' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:35:in select_by_jobs_count’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:25:in block in determine_proxy' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:23:in each’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/services/foreman_tasks/proxy_selector.rb:23:in determine_proxy' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/lib/actions/remote_execution/run_host_job.rb:29:in plan’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:486:in `block (3 levels) in execute_plan’

…
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch' | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in block in create_worker’
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in block in create_with_logging_context' 2018-05-03 15:53:32 [background] [E] The only applicable proxy Puppet-Proxy is down (RuntimeError) | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/lib/actions/remote_execution/run_host_job.rb:33:in plan’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:486:in block (3 levels) in execute_plan' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:35:in plan’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in pass’
…
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch' | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in block in create_worker’
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context’
2018-05-03 15:53:32 [app] [I] Current user: admin (administrator)
2018-05-03 15:53:33 84de2609 [app] [I] Started GET “/job_invocations/30?hosts_needs_refresh=&host_ids_needing_name_update%5B%5D=10&host_ids_needing_status_update%5B%5D=10&=1525355611562" for 10.22.42.88 at 2018-05-03 15:53:33 +0200
2018-05-03 15:53:33 84de2609 [app] [I] Processing by JobInvocationsController#show as JS
2018-05-03 15:53:33 84de2609 [app] [I] Parameters: {“hosts_needs_refresh”=>“”, “host_ids_needing_name_update”=>[“10”], “host_ids_needing_status_update”=>[“10”], "”=>“1525355611562”, “id”=>“30”}
2018-05-03 15:53:33 84de2609 [app] [I] Current user: admin (administrator)
2018-05-03 15:53:33 84de2609 [app] [I] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/show.js.erb
2018-05-03 15:53:33 84de2609 [app] [I] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_host_name_td.html.erb (1.0ms)

A relevant message may be :

The only applicable proxy Puppet-Proxy is down (RuntimeError)

The execution log is:

Action: Actions::RemoteExecution::RunHostsJob
 Entrée: {"job_invocation"=>{"id"=>30, "name"=>"Commands", "description"=>"Run uname"},
 "job_category"=>"Commands",
 "job_invocation_id"=>30,
 "current_user_id"=>4}

 Output: {"planned_count"=>1,
 "cancelled_count"=>0,
 "total_count"=>1,
 "failed_count"=>1,
 "pending_count"=>0,
 "success_count"=>0}

 Exception: RuntimeError: A sub task failed
 Trace de débogage: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/with_sub_plans.rb:230:in `check_for_errors!'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/with_sub_plans.rb:136:in `try_to_finish'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/with_polling_sub_plans.rb:18:in `poll'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/with_polling_sub_plans.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/lib/actions/remote_execution/run_hosts_job.rb:66:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:531:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:31:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:31:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/execution_wrapper.rb:85:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/progress.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/keep_current_user.rb:28:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/keep_current_user.rb:51:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/keep_current_user.rb:28:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:31:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:530:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:529:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:529:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:444:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:444:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:444:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:524:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:277:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/director.rb:55:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/executors/parallel/worker.rb:12:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/executors.rb:11:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

in proxy.log there is this warning

W, [2018-05-03T15:33:12.737781 bd2654e0] WARN – : Puppet server classes cache is disabled, classes retrieval can be slow.

in messages i’ve got

May 3 15:53:32 scvmpr15 smart-proxy: scvmpr15.fr.xxx.priv - - [03/May/2018:15:53:32 CEST] “GET /dynflow/tasks/count?state=running HTTP/1.1” 404 27
May 3 15:53:32 scvmpr15 smart-proxy: - → /dynflow/tasks/count?state=running

regarding the smart proxy i have
•Logs
•Puppet
•Puppet CA
•SSH
•TFTP

in the log part i have:

the file /etc/foreman-proxy/settings.d/dynflow.yml.exists

:enabled: true
:database: /var/lib/foreman-proxy/dynflow/dynflow.sqlite
:core_url: ‘http://127.0.0.1:8008’

BUT dynflow.sqlite does not exist ! and port 8008 is not opened.

the foreman install rpm ( rpm -qa | grep fore)

tfm-rubygem-foreman-tasks-core-0.2.4-1.fm1_17.el7.noarch
foreman-debug-1.17.0-1.el7.noarch
foreman-release-1.18.0-0.develop.201805011537gitd9ea473.el7.noarch
tfm-rubygem-foreman_ansible-2.0.4-1.fm1_18.el7.noarch
tfm-rubygem-foreman_remote_execution-1.4.5-2.fm1_17.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.12.0-2.el7.noarch
foreman-proxy-1.17.0-1.el7.noarch
foreman-installer-1.17.0-1.el7.noarch
foreman-release-scl-4-1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.1.0-1.fm1_17.el7.noarch
foreman-1.17.0-1.el7.noarch
tfm-rubygem-foreman_ansible_core-2.0.2-1.fm1_17.el7.noarch
tfm-rubygem-foreman-tasks-0.11.2-1.fm1_17.el7.noarch
foreman-selinux-1.17.0-1.el7.noarch
foreman-postgresql-1.17.0-1.el7.noarch
foreman-cli-1.17.0-1.el7.noarch

I have no idea where to start searching.
any help is welcome.
thanks
marc

aruzicka · May 3, 2018, 2:27pm

Hello, is the smart_proxy_dynflow_core service running? How did you set up this instance?

tybreizh29 · May 3, 2018, 3:16pm

Hello
tahnks you for your return
the service was not started and not configured also, i enabled it and reboot the machine.
I ended with an Authentication failure

[2018-05-03 17:02:46.635 #3429] ERROR – dynflow: Error initializing command - Net::SSH::AuthenticationFailed Authentication failed for user root@scvmpr15.fr.xxx.priv:

I guess I missed something in the settings file, but i don’t see what

 cat settings.yml
---
# Path to dynflow database, leave blank for in-memory non-persistent database
:database:

# URL of the foreman, used for reporting back
:foreman_url: 'https://scvmpr15.fr.mbda.priv'

# SSL settings for client authentication against Foreman
:foreman_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:foreman_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/scvmpr15.fr.mbda.priv.pem
:foreman_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/scvmpr15.fr.mbda.priv.pem

:console_auth: false

# Set to true to make the core fork to background after start
# :daemonize: false
# :pid_file: /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

# Listen on address
:listen: 127.0.0.1

# Listen on port
:port: 8008

# SSL settings for running core as https service
#:use_https: true
# :ssl_ca_file: ssl/ca.pem
# :ssl_private_key: ssl/localhost.pem
# :ssl_certificate: ssl/certs/localhost.pem
#:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
#:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/scvmpr15.fr.mbda.priv.pem
#:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/scvmpr15.fr.mbda.priv.pem




# Use this option only if you need to disable certain cipher suites.
# Note: we use the OpenSSL suite name, take a look at:
# https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES
# for more information.
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

# Use this option only if you need to strictly specify TLS versions to be
# disabled. SSLv3 and TLS v1.0 are always disabled and cannot be configured.
# Specify versions like: '1.1', or '1.2'
#:tls_disabled_versions: []

# File to log to, leave empty for logging to STDOUT
# :log_file: /var/log/foreman-proxy/smart_proxy_dynflow_core.log

# Log level, one of UNKNOWN, FATAL, ERROR, WARN, INFO, DEBUG
# :log_level: ERROR

# Maximum age of execution plans to keep before having them cleaned
# by the execution plan cleaner (in seconds), defaults to 24 hours
# :execution_plan_cleaner_age: 86400

in …/foreman-proxy/settings.yml
i’ve got

:foreman_url: https://scvmpr15.fr.xxx.priv

# SSL settings for client authentication against Foreman. If undefined, the values
# from general SSL options are used instead. Mainly useful when Foreman uses
# different certificates for its web UI and for smart-proxy requests.
:foreman_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:foreman_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/scvmpr15.fr.xxx.priv.pem
:foreman_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/scvmpr15.fr.xxx.priv.pem

any hint ?

aruzicka · May 3, 2018, 3:25pm

This means there was a problem authenticating against the target host. REX uses a key located at ~foreman-proxy/.ssh/id_rsa_foreman_proxy (on the smart-proxy machine) to authenticate. To allow access to remote machines, you have to deploy corresponding public key to those machines. Please refer to the documentation[1] on how to do it.

[1] - Foreman :: Plugin Manuals

hartmel · May 3, 2018, 6:15pm

Maybe unrelated to your problem : Bug #23482: puppetrun ssh provider does not work without sudo because foreman-proxy user does not have SHELL environment variable - Foreman

tybreizh29 · May 4, 2018, 6:55am

I made it work, but my bad.
i did not see that the keys in /usr/share/foreman-proxy/.ssh/ did belong to root not to foreman-proxy
the command

sudo -u foreman-proxy ssh -i /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy root@xxx

showed it to me.
Thank you very much for your help.