Cannot perform Remote Execution on migrated CentOS 7 Machines

Problem:
Cannot send commands to a CentOS7 machine via Remote Execution. This machine was migrated from Spacewalk to foreman using the bootstrap script, and enrolled to foreman successfully. The public key from the foreman-proxy has been copied to ~/.ssh/authorized_keys properly, and I see no difference in its configuration compared to machines that have not been migrated.

However, when I try a simple yum command via Remote execution, I receive the following error message within the Foreman GUI:

Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried publickey

Expected outcome:

For Remote Execution to work as it does for machines that have been subscribed to foreman from the start

Foreman and Proxy versions:

3.4

Foreman and Proxy plugin versions:

Remote Execution version 8

Distribution and version:

Almalinux 8.6

Other relevant data:

Please let me know if more information is needed.

Thank You

Hi, I’m not familiar with spacewalk at all so I can’t really speak to that, I can only pick on the bits you’ve told us.

The public key from the foreman-proxy has been copied to ~/.ssh/authorized_keys properly,

To ~/.ssh/authorized_keys of what user? Are the permissions on that file correct?

If you bump log_level to DEBUG in /etc/foreman-proxy/settings.yml (or is it .yaml?), restart foreman-proxy and run a job again, /var/log/foreman-proxy/proxy.log will contain the exact ssh commands that are being executed so you can follow along and hopefully it will get you somewhere.

Hi @aruzicka, thanks for the quick response.

The public key was placed in the root users /.ssh/authorized_keys directory

The permissions appear to be correct, and are the same as what I see on a working machine

Thanks for the logging suggestion. I’m going to make that change and run the job again, hopefully more verbose logging will help me narrow this down

I set log_level to debug, restarted foreman-proxy, and attempted to run again.

I didn’t see anything different in the logs.

This is what shows up for me, I can’t make sense of it:

/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matchers/abstract.rb:74:in assigns' /usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:56:in match_value’
/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:36:in block in match?' /usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:35:in each’
/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:35:in match?' /usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:23:in match’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/error.rb:13:in new' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:512:in set_error’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:475:in error!' /usr/share/gems/gems/foreman_remote_execution-8.0.0/app/lib/actions/remote_execution/run_host_job.rb:101:in check_exit_status’
/usr/share/gems/gems/foreman_remote_execution-8.0.0/app/lib/actions/remote_execution/run_host_job.rb:72:in finalize' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:604:in block (2 levels) in execute_finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:40:in finalize' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:20:in block in finalize' /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/execution_wrapper.rb:91:in wrap’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:19:in finalize' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:23:in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/load_setting_values.rb:25:in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:19:in block in finalize’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:19:in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:19:in block in finalize’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:19:in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:25:in block in finalize’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:25:in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in block in finalize’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/world.rb:31:in execute’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:603:in block in execute_finalize' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in block in with_error_handling’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in catch' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in with_error_handling’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:602:in execute_finalize' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:296:in execute’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:78:in run_step’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:63:in dispatch' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:70:in block in run_in_sequence’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:70:in all?' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:70:in run_in_sequence’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:59:in dispatch' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:28:in block in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:48:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:48:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:48:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:48:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:48:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:48:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:29:in block in finalize_phase' /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user’
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:29:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in pass' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in pass’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:48:in finalize_phase' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in call’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/world.rb:31:in execute' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director/sequential_manager.rb:27:in finalize’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director.rb:143:in execute' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors.rb:18:in run_user_code' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in execute_job’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in invoke' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in block in process’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in local’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in block in module:Sidekiq
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in stats’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in call’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in global’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in block in dispatch' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in with_context’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in with_job_hash_context' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in dispatch’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in process' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in process_one’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in run' /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in watchdog’
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in block in safe_thread' /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

Those seem to be foreman logs, not foreman-proxy logs

I pulled these directly from /var/log/foreman-proxy/proxy.log

If that is the case then either then something fairly non standard is going on.

While those entries are in /var/log/foreman-proxy/proxy.log, I was able to see some new log entries when testing this morning that may give us more to work with:

2022-11-08T08:43:47 [D] refresh runner abbea0ac-7991-4a78-aea8-4aa73dff3bdc
2022-11-08T08:43:47 [D] refreshing runner
2022-11-08T08:43:47 [D] finish runner abbea0ac-7991-4a78-aea8-4aa73dff3bdc
2022-11-08T08:43:47 [D] closing session for command [abbea0ac-7991-4a78-aea8-4aa73dff3bdc],0 actors left
2022-11-08T08:43:47 [D] terminate abbea0ac-7991-4a78-aea8-4aa73dff3bdc
2022-11-08T08:43:47 [D] Step da4eabf5-5e4f-481a-af20-ca53752936bc: 3 got event #Proxy::Dynflow::Runner::Update:0x0000559f1f328728
2022-11-08T08:43:47 [D] Step da4eabf5-5e4f-481a-af20-ca53752936bc: 3 suspended >> running in phase Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2022-11-08T08:43:47 69cee379 [D] Step da4eabf5-5e4f-481a-af20-ca53752936bc: 3 running >> success in phase Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2022-11-08T08:43:47 [D] Step da4eabf5-5e4f-481a-af20-ca53752936bc: 6 pending >> running in phase Run Proxy::Dynflow::Callback::Action
2022-11-08T08:43:47 69cee379 [D] Step da4eabf5-5e4f-481a-af20-ca53752936bc: 6 running >> success in phase Run Proxy::Dynflow::Callback::Action
2022-11-08T08:43:47 [D] Step da4eabf5-5e4f-481a-af20-ca53752936bc: 4 pending >> running in phase Finalize Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2022-11-08T08:43:47 69cee379 [E] Script execution failed
2022-11-08T08:43:47 69cee379 [D] Step da4eabf5-5e4f-481a-af20-ca53752936bc: 4 running >> error in phase Finalize Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2022-11-08T08:43:47 [D] ExecutionPlan da4eabf5-5e4f-481a-af20-ca53752936bc running >> stopped
2022-11-08T08:43:53 [D] Executor heartbeat
2022-11-08T08:43:57 [D] Step 8fe61bbc-9ae3-4cda-8a9b-662cc7d550cf: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2022-11-08T08:43:57 [D] Step 8fe61bbc-9ae3-4cda-8a9b-662cc7d550cf: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2022-11-08T08:43:57 69cee379 [E] A sub task failed

So I ended up finding the solution via another forum post here suggesting I try:

sudo -u foreman-proxy ssh root@foreman.example.com -i https://localhost:8443/ssh/pubkey

I received an immediate permission denied error

My permissions for ~/.ssh and ~/authorized_keys were correct, so I checked /etc/sshd/sshd_config and changed it to both allow root login and password authentication.

Remote execution via Foreman GUI was still not happy. I tested ssh again via Foreman shell, and monitored the status of sshd.service on the host I was trying to connect to.

I received a log stating “Authentication refused: bad ownership or modes for directory /root”

As it turns out, someone had changed permissions to the entire root directory on this particular CentOS 7 machine I migrated.

Once I changed ownership back to root on this directory, all was happy with remote execution from within the GUI.

If I had picked any other CentOS 7 to test, I probably wouldn’t have run into this issue :slight_smile:

On a sidenote, in case someone comes across this thread. The command you have given is not correct and it doesn’t make sense to use -i with that URL. I don’t even know if that’s possible to use an URL. Either way, loading a public key as identity is pointless. You need a private key, not a public key to connect. The server you connect to needs the public key in the authorized_keys file.

The correct command would be

sudo -u foreman-proxy ssh root@client.example.com -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy

That’s what remote execution runs (unless you have made some changes in settings, e.g. to use sudo on the client instead of direct root login)…

For testing, it’s usually a good idea to add -v to the ssh command to see what is going on exactly…