Foreman REX authenticatin issue

Support
,
1

Problem:
We are facing a strange issue with REX after upgrade from 2.0 to 2.1, 2.2, ansible is working fine, the same credentials are not working and throwing error like NET: SSH authentication failed.

ERROR Log:

172.25.0.2 - - [25/Sep/2020:10:23:52 IST] "GET /tasks/count?state=running HTTP/1.1" 200 29
172.25.0.2 - - [25/Sep/2020:10:23:53 IST] "POST /tasks/launch? HTTP/1.1" 200 110
error while initalizing command Net::SSH::AuthenticationFailed Authentication failed for user root@10.10.153.35:
 /usr/share/gems/gems/net-ssh-5.1.0/lib/net/ssh.rb:259:in `start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:273:in `session'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:344:in `run_sync'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:435:in `ensure_remote_directory'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:411:in `upload_data'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:407:in `cp_script_to_remote'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:172:in `prepare_start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:158:in `start'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:32:in `start_runner'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:7:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:24:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:106:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `block in on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:59:in `with_backtrace'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:353:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:342:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `block in create_worker'Error initializing command - Net::SSH::AuthenticationFailed Authentication failed for user root@10.10.153.35:
/usr/share/gems/gems/net-ssh-5.1.0/lib/net/ssh.rb:259:in `start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:273:in `session'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:344:in `run_sync'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:435:in `ensure_remote_directory'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:411:in `upload_data'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:407:in `cp_script_to_remote'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:172:in `prepare_start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:158:in `start'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:32:in `start_runner'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:7:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:24:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:106:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `block in on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:59:in `with_backtrace'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:353:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:342:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `block in create_worker'Script execution failedA sub task failed (RuntimeError)
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:231:in `check_for_errors!'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:137:in `try_to_finish'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_polling_sub_plans.rb:19:in `poll'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_polling_sub_plans.rb:11:in `run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:106:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `block in on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:59:in `with_backtrace'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:353:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:342:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `block in create_worker'

After upgrade from 2.0 to 2.1

Expected outcome:

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

2

same credentials

as in username and password?

3

@aruzicka, I’m using the same credentials for both ansible and REX. But the authentication problem is happening for only REX.

4

I understood that you are using the same mechanism for authentication, I was asking whether you use username and password or username and private+public key

5

Hi Aruzicka,
We are using a username and password for authentication.

6

And should it be connecting as root or as a different user? I’m trying to find out if the issue is the password or the username. From the stack trace it currently tries to log in as root

7

Hi,

Pleae find the detaild logs for ansible and REX

REX Log:

/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in
block in create_worker'Error initializing command - Net::SSH::AuthenticationFailed Authentication failed for user usertest@10.10.153.31: /usr/share/gems/gems/net-ssh-5.1.0/lib/net/ssh.rb:259:in start’

REX SSHD debug log:

2020-09-25T15:27:03.822526+03:00 sapbhpv sshd[10816]: debug1: temporarily_use_uid: 1009/100 (e=0/0)
2020-09-25T15:27:03.822804+03:00 sapbhpv sshd[10816]: debug1: trying public key file /home/usertest/.ssh/authorized_keys
2020-09-25T15:27:03.823063+03:00 sapbhpv sshd[10816]: debug1: Could not open authorized keys ‘/home/usertest/.ssh/authorized_keys’: No such file or directory
2020-09-25T15:27:03.823321+03:00 sapbhpv sshd[10816]: debug1: restore_uid: 0/0
2020-09-25T15:27:03.823570+03:00 sapbhpv sshd[10816]: Failed publickey for usertest from 10.10.146.254 port 59754 ssh2: RSA SHA256:Rgn7vuqEO44/wFybSsWFy48FA2C7KXd3anL/URBZP4o
2020-09-25T15:27:03.823824+03:00 sapbhpv sshd[10816]: Connection closed by 10.10.146.254 port 59754 [preauth]

With ansible Log:

Parameters: {“callbacks”=>[{“callback”=>{“task_id”=>“4c017c9e-3108-4732-887a-8dc81dce80b0”, “step_id”=>3},
“data”=>{“result”=>[{“output_type”=>“stdout”, “output”=>“\n”, “timestamp”=>1601036396.9140766}, {“output_type”=>“stdout”,
“output”=>“\r\nPLAY [all] *********************************************************************\n”, “timestamp”=>1601036396.9142642},
{“output_type”=>“stdout”, “output”=>“\r\nTASK [Gathering Facts] *********************************************************\n”, “timestamp”=>1601036396.9144855},
{“output_type”=>“stdout”, “output”=>“\n”, “timestamp”=>1601036396.91 46252}, {“output_type”=>“stdout”, “output”=>“[WARNING]:
Platform linux on host test-bo-c-prod–db-node-10.10.153.31 is\n”,“timestamp”=>1601036399.0273337}, {“output_type”=>“stdout”,
“output”=>“using the discovered Python interpreter at /usr/bin/py thon, but future\n”, “timestamp”=>1601036399.027481}, {“output_type”=>“stdout”,
“output”=>“installation of another Python interpreter could change this. See https://docs.\n”,
“timestamp”=>1601036399.0275943}, {“output_type”=>“stdout”, “output”=>“ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for\n”,
“timestamp”=>1601036399.02767}, {“output_type”=>“stdout”, “output”=>“more information.\n”, “timestamp”=>1601036399.0277874}, {“output_type”=>“stdout”,
“output”=>“ok: [test-bo-c-prod–db-node-10.10.153.31]\n”, “timestamp”=>1601036399.036786}, {“output_type”=>“stdout”, “output”=>“\r\nTASK [shell] ****
***************************************************************\n”, “timestamp”=>1601036401.1483483}, {“output_type”=>“stdout”
, “output”=>“\n”, “timestamp”=>1601036401.148512}, {“output_type”=>“stdout”, “output”=>“changed:
[test-bo-c-prod–db-node- 10.10.153.31]\n”, “timestamp”=>1601036402.1528656}, {“output_type”=>“stdout”, “output”=>"\r\nTASK [debug] ********************

Hostgroup Parameters:

image
image906×276 7.12 KB

8

@aruzicka, Thanks for responding to the request, my security scanner was considered batch scheduling as ssh flooding, due to proxy IP got blacklisted. Now issue got resolved after whitelisting the IP in the scanner.