Remote action failed

Support
,
1

Problem:

Hello,

I’m a beginner on the product and i’m trying to perform a remote action on a test VM (name: test-frm). The remote action is pretty simple:

Run yum check-update on test-vm

Unfortunatly, the job failed with the message error:

1:Error initializing command: Net::SSH::Exception - could not settle on hmac_client algorithm
2:Exit status: EXCEPTION

Thank you for your help

Expected outcome:

A successfull yum check-update

Foreman and Proxy versions:

  • Foreman v2.5.0

Active features

Dynflow v0.3.0
Logs
Pulpcore 3.0.0
Puppet 2.5.0
Puppet CA 2.5.0
Registration 2.5.0
SSH 0.3.1

Foreman and Proxy plugin versions:

  • Foreman-tasks v4.1.1
  • Foreman_remote_execution v4.5.0
  • Katello v4.1.0.rc2.1

Distribution and version:

Both Foreman and the test vm are in CentOS 7: 7.9.2009

Other relevant data:

Error logs from the task:

Action:

Actions::RemoteExecution::RunHostJob

Input:

{“host”=>{“id”=>2, “name”=>“test-frm.mydomain”},
“job_category”=>“Commands”,
“description”=>“Run yum check-update”,
“job_invocation_id”=>8,
“job_features”=>,
“delegated_action_id”=>2,
“current_request_id”=>“5a3b895c-e60d-4c9d-98c7-9d01695bb36d”,
“current_timezone”=>“Europe/Paris”,
“current_user_id”=>4,
“current_organization_id”=>1,
“current_location_id”=>2}

Output:

{}

Exception:

StandardError: Job execution failed

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in block in assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in tap’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in match_value’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in block in match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in match’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/error.rb:13:in new' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:501:in set_error’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:464:in error!' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.5.0/app/lib/actions/remote_execution/run_host_job.rb:94:in check_exit_status’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.5.0/app/lib/actions/remote_execution/run_host_job.rb:65:in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:593:in block (2 levels) in execute_finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:40:in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:20:in block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:19:in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:23:in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_request_id.rb:19:in block in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_request_id.rb:19:in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_timezone.rb:19:in block in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_timezone.rb:19:in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_user.rb:25:in block in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_user.rb:25:in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in block in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:592:in block in execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:591:in execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:78:in run_step’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:63:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:70:in block in run_in_sequence’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:70:in all?' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:70:in run_in_sequence’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:59:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:28:in block in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:48:in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:48:in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:48:in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:48:in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:48:in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_user.rb:29:in block in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.1/app/lib/actions/middleware/keep_current_user.rb:29:in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:48:in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director/sequential_manager.rb:27:in finalize’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:122:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

2

Hi,
judging from the netssh exception the ssh client and server cannot agree on a crypto algorithm to use to secure the communication. What os is the target machine running? Is it running in some special mode like fips?

4

Hi,

It’s the same as foreman server (Cent OS 7).

After checking, FIPS is not enable :

$ cat /proc/sys/crypto/fips_enabled
0

If I open an ssh session from my foreman to my test-vm, it’s okay. I have no issue

5

I can’t say I’ve seen this one. Could you check logs of ssh server on the target machine with increased verbosity?

6

Did you customize any ssh server configs? Did you modify the cryptopolicy? If so, what does update-crypto-policies --show show? Do you use FreeIPA or similar on any of these machines? Did you try to set a different connection user than root? Can you also try sudo -u foreman-proxy ssh $target_vm -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy to better simulate for foreman-proxy does? Does it print any error?

7

Hello, sorry for my late answer. update-crypto policies show nothing (command not found).

We do not use FreeIPA or anything else, when I tried to perform " sudo -u foreman-proxy ssh $target_vm -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy`" it ask the password (I do not know this one).

8

Heere the log:

Jun 15 10:33:06 test-vm sshd[50201]: debug1: Set /proc/self/oom_score_adj from 0 to -1000
Jun 15 10:33:06 test-vm sshd[50201]: debug2: fd 3 setting O_NONBLOCK
Jun 15 10:33:06 test-vm sshd[50201]: debug1: Bind to port 22 on 0.0.0.0.
Jun 15 10:33:06 test-vm sshd[50201]: Server listening on 0.0.0.0 port 22.
Jun 15 10:33:06 test-vm sshd[50201]: debug2: fd 4 setting O_NONBLOCK
Jun 15 10:33:06 test-vm sshd[50201]: debug1: Bind to port 22 on ::.
Jun 15 10:33:06 test-vm sshd[50201]: Server listening on :: port 22.
Jun 15 10:33:21 test-vm sshd[50201]: debug1: Forked child 50216.
Jun 15 10:33:21 test-vm sshd[50216]: debug1: Set /proc/self/oom_score_adj to 0
Jun 15 10:33:21 test-vm sshd[50216]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jun 15 10:33:21 test-vm sshd[50216]: debug1: inetd sockets after dupping: 3, 3
Jun 15 10:33:21 test-vm sshd[50216]: Connection from xxx.xxx.xxx.xxx port 42934 on xxx.xxx.xxx.xxx port 22
Jun 15 10:33:21 test-vm sshd[50216]: debug1: Client protocol version 2.0; client software version Ruby/Net::SSH_4.2.0 x86_64-linux
Jun 15 10:33:21 test-vm sshd[50216]: debug1: no match: Ruby/Net::SSH_4.2.0 x86_64-linux
Jun 15 10:33:21 test-vm sshd[50216]: debug1: Local version string SSH-2.0-OpenSSH_7.4
Jun 15 10:33:21 test-vm sshd[50216]: debug1: Enabling compatibility mode for protocol 2.0
Jun 15 10:33:21 test-vm sshd[50216]: debug2: fd 3 setting O_NONBLOCK
Jun 15 10:33:21 test-vm sshd[50216]: debug2: Network child is on pid 50217
Jun 15 10:33:21 test-vm sshd[50216]: debug1: SELinux support disabled [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: permanently_set_uid: 74/74 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: SSH2_MSG_KEXINIT received [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: local server KEXINIT proposal [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: KEX algorithms: curve25519-sha256 ,curve25519-sha256@libssh. org ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sa512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: ciphers ctos: aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: ciphers stoc: aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: MACs ctos: hmac-sha2-512-etm@ openssh. com,hmac-sha2-256-etm@openssh. com [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: MACs stoc: hmac-sha2-512-etm@ openssh. com, hmac-sha2-256-etm@openssh. com [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: compression ctos: none,zlib@openssh. com [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: compression stoc: none,zlib@openssh. com [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: languages ctos: [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: languages stoc: [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: first_kex_follows 0 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: reserved 0 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: peer client KEXINIT proposal [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: KEX algorithms: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,cdh-sha2-nistp521 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: host key algorithms: ssh-rsa,ssh-dss,ssh-rsa-cert-v01@openssh. com,ssh-rsa-cert-v00@openssh. com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: ciphers ctos: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,idea-cbc,arcfour128,arcfour256,arcfour,aes128-ctr,aes192-ctr,aes256-ctr,ast128-ctr,blowfish-ctr,3des-ctr,none [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: ciphers stoc: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,idea-cbc,arcfour128,arcfour256,arcfour,aes128-ctr,aes192-ctr,aes256-ctr,ast128-ctr,blowfish-ctr,3des-ctr,none [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: MACs ctos: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-ripemd160,hmac-ripemd160@openssh. com,hmac-sha2-256,hmac-sha2-512,hmac-sha2-256-96,hmac-sha2-512-96,none [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: MACs stoc: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-ripemd160,hmac-ripemd160@openssh. com,hmac-sha2-256,hmac-sha2-512,hmac-sha2-256-96,hmac-sha2-512-96,none [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: compression ctos: none,zlib@openssh. com,zlib [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: compression stoc: none,zlib@openssh. com,zlib [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: languages ctos: [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: languages stoc: [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: first_kex_follows 0 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug2: reserved 0 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: kex: algorithm: diffie-hellman-group-exchange-sha1 [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: kex: host key algorithm: ssh-rsa [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: Unable to negotiate with xxx.xxx.xxx.xxx port 42934: no matching MAC found. Their offer: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-ripemd160,hmac-ripemd160@openssh. com,hmac-sha2-256,mac-sha2-512,hmac-sha2-256-96,hmac-sha2-512-96,none [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: do_cleanup [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: monitor_read_log: child log fd closed
Jun 15 10:33:21 test-vm sshd[50216]: debug1: do_cleanup
Jun 15 10:33:21 test-vm sshd[50216]: debug1: Killing privsep child 50217
leanup [preauth]
Jun 15 10:33:21 test-vm sshd[50216]: debug1: monitor_read_log: child log fd closed
Jun 15 10:33:21 test-vm sshd[50216]: debug1: do_cleanup
Jun 15 10:33:21 test-vm sshd[50216]: debug1: Killing privsep child 50217
[/details]](https://)

9

That cannot be quite right: you have posted the server log of a connection which was not possible because of no matching MAC found. There would be no password question on the ssh side because it doesn’t even get that far.

Run ssh again and add -v for verbose output. Then collect the matching ssh -v and sshd log output.

10

I posted the log that [aruzicka] asked for. So yes, no link with the user foreman-proxy.

11

Well, the server refuses the connection because it cannot negotiate a mac algorithm. It would be helpful to have a matching set of client and server logs of the same, failing connection to see both sides.

On the foreman server run

sudo -u foreman-proxy ssh -vvv -l root $target_vm -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy

if you have set up the root account on the client side for remote execution. If you use a different account with sudo rights, use that.

Capture the log of ssh on the foreman server and the sshd log on the target server from that attempt.

Are you able at all to log into the target server from the foreman server? That is log into the foreman server and then use ssh to log into the target server with any account you can use there.

Also run

$ ssh -Q mac
hmac-sha1
...

on both side (foreman and target server) and also check the ssh and sshd configuration for the configuration for “MACs”, i.e. /etc/ssh/ssh_config on the foreman server and /etc/ssh/sshd_config on the target server, if there is anything particular set up.

12

It’s weir, when i perform an ssh, i do not have any errors:

Log from foreman:

$ sudo -u foreman-proxy ssh -vvv -l admin test-vm -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving “test-vm” port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to test-vm [test-vm] port 22.
debug1: Connection established.
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 1
debug1: key_load_public: No such file or directory
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to test-vm:22 as ‘admin’
debug3: hostkeys_foreach: reading file “/usr/share/foreman-proxy/.ssh/known_hosts”
debug3: record_hostkey: found key type ECDSA in file /usr/share/foreman-proxy/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from test-vm
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh. com,ecdsa-sha2-nistp384-cert-v01@openssh. com,ecdsa-sha2-nistp521-cert-v01@openssh. com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh. org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh. com,ecdsa-sha2-nistp384-cert-v01@openssh. com,ecdsa-sha2-nistp521-cert-v01@openssh. com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh. com,ssh-rsa-cert-v01@openssh. com,ssh-dss-cert-v01@openssh. com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh. com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh. com,aes256-gcm@openssh. com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh. com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh. com,aes256-gcm@openssh. com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh. com,umac-128-etm@openssh. com,hmac-sha2-256-etm@openssh. com,hmac-sha2-512-etm@openssh. com,hmac-sha1-etm@openssh. com,umac-64@openssh. com,umac-128@openssh. com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh. com,umac-128-etm@openssh. com,hmac-sha2-256-etm@openssh. com,hmac-sha2-512-etm@openssh. com,hmac-sha1-etm@openssh. com,umac-64@openssh. com,umac-128@openssh. com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh. com,zlib
debug2: compression stoc: none,zlib@openssh. com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh. org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh. com,hmac-sha2-256-etm@openssh. com
debug2: MACs stoc: hmac-sha2-512-etm@openssh. com,hmac-sha2-256-etm@openssh. com
debug2: compression ctos: none,zlib@openssh. com
debug2: compression stoc: none,zlib@openssh. com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh. com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh. com compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:tWTZeUdStUracwJTb/81CppYMrhJrVxdsuuXGmY4n4I
debug3: hostkeys_foreach: reading file “/usr/share/foreman-proxy/.ssh/known_hosts”
debug3: record_hostkey: found key type ECDSA in file /usr/share/foreman-proxy/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from test-vm
debug1: Host ‘test-vm’ is known and matches the ECDSA host key.
debug1: Found key in /usr/share/foreman-proxy/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy (0x560079331610), explicit
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:992)

debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:992)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:OfMUAtRYCW0hs0C+SdIFNdGV6EUbjo60HDYISIno6xk
debug3: sign_and_send_pubkey: RSA SHA256:OfMUAtRYCW0hs0C+SdIFNdGV6EUbjo60HDYISIno6xk
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to test-vm ([test-vm]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh. com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh. com want_reply 0
debug3: receive packet: type 91
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env HOSTNAME
debug3: Ignored env TERM
debug3: Ignored env HISTSIZE
debug3: Ignored env LS_COLORS
debug3: Ignored env MAIL
debug1: Sending env LC_CTYPE = fr_FR.utf8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env PATH
debug3: Ignored env LOGNAME
debug3: Ignored env USER
debug3: Ignored env USERNAME
debug3: Ignored env HOME
debug3: Ignored env SHELL
debug3: Ignored env SUDO_COMMAND
debug3: Ignored env SUDO_USER
debug3: Ignored env SUDO_UID
debug3: Ignored env SUDO_GID
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env XDG_RUNTIME_DIR
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Tue Jun 22 16:38:56 2021 from xxx.xxx.xxx.xxx
[admin@test-vm ~]$ exitdebug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype eow@openssh. com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed

déconnexion
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: send packet: type 1
Connection to test-vm closed.
Transferred: sent 3320, received 2436 bytes, in 7.3 seconds
Bytes per second: sent 451.9, received 331.6
debug1: Exit status 0

Log from target:

Jun 22 16:41:44 test-vm sshd[2965]: debug1: Forked child 32332.
Jun 22 16:41:44 test-vm sshd[32332]: debug1: Set /proc/self/oom_score_adj to 0
Jun 22 16:41:44 test-vm sshd[32332]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jun 22 16:41:44 test-vm sshd[32332]: debug1: inetd sockets after dupping: 3, 3
Jun 22 16:41:44 test-vm sshd[32332]: Connection from foreman port 40530 on test-vm port 22
Jun 22 16:41:44 test-vm sshd[32332]: debug1: Client protocol version 2.0; client software version OpenSSH_7.4
Jun 22 16:41:44 test-vm sshd[32332]: debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
Jun 22 16:41:44 test-vm sshd[32332]: debug1: Local version string SSH-2.0-OpenSSH_7.4
Jun 22 16:41:44 test-vm sshd[32332]: debug1: Enabling compatibility mode for protocol 2.0
Jun 22 16:41:44 test-vm sshd[32332]: debug2: fd 3 setting O_NONBLOCK
Jun 22 16:41:44 test-vm sshd[32332]: debug2: Network child is on pid 32333
Jun 22 16:41:44 test-vm sshd[32332]: debug1: SELinux support disabled [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: permanently_set_uid: 74/74 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: SSH2_MSG_KEXINIT received [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: local server KEXINIT proposal [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh. org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: ciphers ctos: aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: ciphers stoc: aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: MACs ctos: hmac-sha2-512-etm@openssh. com,hmac-sha2-256-etm@openssh. com [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: MACs stoc: hmac-sha2-512-etm@openssh. com,hmac-sha2-256-etm@openssh. com [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: compression ctos: none,zlib@openssh. com [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: compression stoc: none,zlib@openssh. com [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: languages ctos: [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: languages stoc: [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: first_kex_follows 0 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: reserved 0 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: peer client KEXINIT proposal [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh. org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh. com,ecdsa-sha2-nistp384-cert-v01@openssh. com,ecdsa-sha2-nistp521-cert-v01@openssh. com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh. com,ssh-rsa-cert-v01@openssh. com,ssh-dss-cert-v01@openssh. com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: ciphers ctos: chacha20-poly1305@openssh. com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh. com,aes256-gcm@openssh. com,aes128-cbc,aes192-cbc,aes256-cbc [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: ciphers stoc: chacha20-poly1305@openssh. com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh. com,aes256-gcm@openssh. com,aes128-cbc,aes192-cbc,aes256-cbc [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: MACs ctos: umac-64-etm@openssh. com,umac-128-etm@openssh. com,hmac-sha2-256-etm@openssh. com,hmac-sha2-512-etm@openssh. com,hmac-sha1-etm@openssh. com,umac-64@openssh. com,umac-128@openssh. com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: MACs stoc: umac-64-etm@openssh. com,umac-128-etm@openssh. com,hmac-sha2-256-etm@openssh. com,hmac-sha2-512-etm@openssh. com,hmac-sha1-etm@openssh. com,umac-64@openssh. com,umac-128@openssh. com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: compression ctos: none,zlib@openssh. com,zlib [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: compression stoc: none,zlib@openssh. com,zlib [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: languages ctos: [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: languages stoc: [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: first_kex_follows 0 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: reserved 0 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh. com compression: none [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh. com compression: none [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: kex: curve25519-sha256 need=32 dh_need=32 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: kex: curve25519-sha256 need=32 dh_need=32 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: monitor_read: 6 used once, disabling now
Jun 22 16:41:44 test-vm sshd[32332]: debug2: set_newkeys: mode 1 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: rekey after 4294967296 blocks [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug2: set_newkeys: mode 0 [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: rekey after 4294967296 blocks [preauth]
Jun 22 16:41:44 test-vm sshd[32332]: debug1: KEX done [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: userauth-request for user admin service ssh-connection method none [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: attempt 0 failures 0 [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug2: parse_server_config: config reprocess config len 1085
Jun 22 16:41:45 test-vm sshd[32332]: debug2: monitor_read: 8 used once, disabling now
Jun 22 16:41:45 test-vm sshd[32332]: debug2: input_userauth_request: setting up authctxt for admin [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: PAM: initializing for “admin”
Jun 22 16:41:45 test-vm sshd[32332]: debug1: PAM: setting PAM_RHOST to “foreman.domain”
Jun 22 16:41:45 test-vm sshd[32332]: debug1: PAM: setting PAM_TTY to “ssh”
Jun 22 16:41:45 test-vm sshd[32332]: debug2: monitor_read: 100 used once, disabling now
Jun 22 16:41:45 test-vm sshd[32332]: debug2: monitor_read: 4 used once, disabling now
Jun 22 16:41:45 test-vm sshd[32332]: debug2: monitor_read: 80 used once, disabling now
Jun 22 16:41:45 test-vm sshd[32332]: debug2: monitor_read: 10 used once, disabling now
Jun 22 16:41:45 test-vm sshd[32332]: debug2: input_userauth_request: try method none [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: userauth-request for user admin service ssh-connection method publickey [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: attempt 1 failures 0 [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug2: input_userauth_request: try method publickey [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:OfMUAtRYCW0hs0C+SdIFNdGV6EUbjo60HDYISIno6xk [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
Jun 22 16:41:45 test-vm sshd[32332]: debug1: trying public key file /home/admin/.ssh/authorized_keys
Jun 22 16:41:45 test-vm sshd[32332]: debug1: fd 4 clearing O_NONBLOCK
Jun 22 16:41:45 test-vm sshd[32332]: debug1: matching key found: file /home/admin/.ssh/authorized_keys, line 1 RSA SHA256:OfMUAtRYCW0hs0C+SdIFNdGV6EUbjo60HDYISIno6xk
Jun 22 16:41:45 test-vm sshd[32332]: debug1: restore_uid: 0/0
Jun 22 16:41:45 test-vm sshd[32332]: debug2: userauth_pubkey: authenticated 0 pkalg rsa-sha2-512 [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: Postponed publickey for admin from foreman port 40530 ssh2 [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: userauth-request for user admin service ssh-connection method publickey [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: attempt 2 failures 0 [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug2: input_userauth_request: try method publickey [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
Jun 22 16:41:45 test-vm sshd[32332]: debug1: trying public key file /home/admin/.ssh/authorized_keys
Jun 22 16:41:45 test-vm sshd[32332]: debug1: fd 4 clearing O_NONBLOCK
Jun 22 16:41:45 test-vm sshd[32332]: debug1: matching key found: file /home/admin/.ssh/authorized_keys, line 1 RSA SHA256:OfMUAtRYCW0hs0C+SdIFNdGV6EUbjo60HDYISIno6xk
Jun 22 16:41:45 test-vm sshd[32332]: debug1: restore_uid: 0/0
Jun 22 16:41:45 test-vm sshd[32332]: debug1: do_pam_account: called
Jun 22 16:41:45 test-vm sshd[32332]: Accepted publickey for admin from foreman port 40530 ssh2: RSA SHA256:OfMUAtRYCW0hs0C+SdIFNdGV6EUbjo60HDYISIno6xk
Jun 22 16:41:45 test-vm sshd[32332]: debug1: monitor_child_preauth: admin has been authenticated by privileged process
Jun 22 16:41:45 test-vm sshd[32332]: debug2: userauth_pubkey: authenticated 1 pkalg rsa-sha2-512 [preauth]
Jun 22 16:41:45 test-vm sshd[32332]: debug1: monitor_read_log: child log fd closed
Jun 22 16:41:45 test-vm sshd[32332]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
Jun 22 16:41:45 test-vm sshd[32332]: debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
Jun 22 16:41:45 test-vm sshd[32332]: debug1: restore_uid: 0/0
Jun 22 16:41:45 test-vm sshd[32332]: debug1: SELinux support disabled
Jun 22 16:41:45 test-vm sshd[32332]: debug1: PAM: establishing credentials
Jun 22 16:41:45 test-vm sshd[32332]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jun 22 16:41:45 test-vm sshd[32332]: User child is on pid 32334
Jun 22 16:41:45 test-vm sshd[32265]: debug1: server_input_channel_req: channel 0 request winadj@putty.projects.tartarus. org reply 1
Jun 22 16:41:45 test-vm sshd[32265]: debug1: session_by_channel: session 0 channel 0
Jun 22 16:41:45 test-vm sshd[32265]: debug1: session_input_channel_req: session 0 req winadj@putty.projects.tartarus. org
Jun 22 16:41:45 test-vm sshd[32265]: debug2: channel 0: rcvd adjust 33080
Jun 22 16:41:45 test-vm sshd[32334]: debug1: PAM: establishing credentials
Jun 22 16:41:45 test-vm sshd[32334]: debug1: permanently_set_uid: 1000/1000
Jun 22 16:41:45 test-vm sshd[32334]: debug2: set_newkeys: mode 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: rekey after 4294967296 blocks
Jun 22 16:41:45 test-vm sshd[32334]: debug2: set_newkeys: mode 1
Jun 22 16:41:45 test-vm sshd[32334]: debug1: rekey after 4294967296 blocks
Jun 22 16:41:45 test-vm sshd[32334]: debug1: ssh_packet_set_postauth: called
Jun 22 16:41:45 test-vm sshd[32334]: debug1: Entering interactive session for SSH2.
Jun 22 16:41:45 test-vm sshd[32334]: debug2: fd 4 setting O_NONBLOCK
Jun 22 16:41:45 test-vm sshd[32334]: debug2: fd 7 setting O_NONBLOCK
Jun 22 16:41:45 test-vm sshd[32334]: debug1: server_init_dispatch
Jun 22 16:41:45 test-vm sshd[32334]: debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
Jun 22 16:41:45 test-vm sshd[32334]: debug1: input_session_request
Jun 22 16:41:45 test-vm sshd[32334]: debug1: channel 0: new [server-session]
Jun 22 16:41:45 test-vm sshd[32334]: debug2: session_new: allocate (allocated 0 max 10)
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_new: session 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_open: channel 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_open: session 0: link with channel 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: server_input_channel_open: confirm session
Jun 22 16:41:45 test-vm sshd[32334]: debug1: server_input_global_request: rtype no-more-sessions@openssh. com want_reply 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: server_input_channel_req: channel 0 request pty-req reply 1
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_by_channel: session 0 channel 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_input_channel_req: session 0 req pty-req
Jun 22 16:41:45 test-vm sshd[32334]: debug1: Allocating pty.
Jun 22 16:41:45 test-vm sshd[32332]: debug2: session_new: allocate (allocated 0 max 10)
Jun 22 16:41:45 test-vm sshd[32332]: debug1: session_new: session 0
Jun 22 16:41:45 test-vm sshd[32332]: debug1: SELinux support disabled
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_pty_req: session 0 alloc /dev/pts/0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: server_input_channel_req: channel 0 request env reply 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_by_channel: session 0 channel 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_input_channel_req: session 0 req env
Jun 22 16:41:45 test-vm sshd[32334]: debug2: Setting env 0: LC_CTYPE=fr_FR.utf8
Jun 22 16:41:45 test-vm sshd[32334]: debug1: server_input_channel_req: channel 0 request shell reply 1
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_by_channel: session 0 channel 0
Jun 22 16:41:45 test-vm sshd[32334]: debug1: session_input_channel_req: session 0 req shell
Jun 22 16:41:45 test-vm sshd[32334]: Starting session: shell on pts/0 for admin from foreman port 40530 id 0
Jun 22 16:41:45 test-vm sshd[32334]: debug2: fd 3 setting TCP_NODELAY
Jun 22 16:41:45 test-vm sshd[32334]: debug2: channel 0: rfd 12 isatty
Jun 22 16:41:45 test-vm sshd[32334]: debug2: fd 12 setting O_NONBLOCK
Jun 22 16:41:45 test-vm sshd[32335]: debug1: Setting controlling tty using TIOCSCTTY.
Jun 22 16:41:52 test-vm sshd[32334]: debug1: Received SIGCHLD.
Jun 22 16:41:52 test-vm sshd[32334]: debug1: session_by_pid: pid 32335
Jun 22 16:41:52 test-vm sshd[32334]: debug1: session_exit_message: session 0 channel 0 pid 32335
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: request exit-status confirm 0
Jun 22 16:41:52 test-vm sshd[32334]: debug1: session_exit_message: release channel 0
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: write failed
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: close_write
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: send eow
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: output open -> closed
Jun 22 16:41:52 test-vm sshd[32334]: debug2: notify_done: reading
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: read<=0 rfd 12 len -1
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: read failed
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: close_read
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: input open -> drain
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: ibuf empty
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: send eof
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: input drain -> closed
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: send close
Jun 22 16:41:52 test-vm sshd[32332]: debug1: session_by_tty: session 0 tty /dev/pts/0
Jun 22 16:41:52 test-vm sshd[32332]: debug1: session_pty_cleanup: session 0 release /dev/pts/0
Jun 22 16:41:52 test-vm sshd[32334]: debug2: channel 0: rcvd close
Jun 22 16:41:52 test-vm sshd[32334]: Received disconnect from foreman port 40530:11: disconnected by user
Jun 22 16:41:52 test-vm sshd[32334]: Disconnected from foreman port 40530
Jun 22 16:41:52 test-vm sshd[32334]: debug1: do_cleanup
Jun 22 16:41:52 test-vm sshd[32332]: debug1: do_cleanup
Jun 22 16:41:52 test-vm sshd[32332]: debug1: PAM: cleanup
Jun 22 16:41:52 test-vm sshd[32332]: debug1: PAM: closing session
Jun 22 16:41:52 test-vm sshd[32332]: pam_unix(sshd:session): session closed for user admin
Jun 22 16:41:52 test-vm sshd[32332]: debug1: PAM: deleting credentials

ssh -Q mac résult:

Foreman:
ssh -Q mac
hmac-sha1
hmac-sha1-96
hmac-sha2-256
hmac-sha2-512
hmac-md5
hmac-md5-96
hmac-ripemd160
hmac-ripemd160@openssh. com
umac-64@openssh. com
umac-128@openssh. com
hmac-sha1-etm@openssh. com
hmac-sha1-96-etm@openssh. com
hmac-sha2-256-etm@openssh. com
hmac-sha2-512-etm@openssh. com
hmac-md5-etm@openssh. com
hmac-md5-96-etm@openssh. com
hmac-ripemd160-etm@openssh. com
umac-64-etm@openssh. com
umac-128-etm@openssh. com

Taget VM:

ssh -Q mac
hmac-sha1
hmac-sha1-96
hmac-sha2-256
hmac-sha2-512
hmac-md5
hmac-md5-96
hmac-ripemd160
hmac-ripemd160@openssh. com
umac-64@openssh. com
umac-128@openssh. com
hmac-sha1-etm@openssh. com
hmac-sha1-96-etm@openssh. com
hmac-sha2-256-etm@openssh. com
hmac-sha2-512-etm@openssh. com
hmac-md5-etm@openssh. com
hmac-md5-96-etm@openssh. com
hmac-ripemd160-etm@openssh. com
umac-64-etm@openssh. com
umac-128-etm@openssh. com

13

I think you hit an issue possibly relating to openscap / crypto defaults on newer OSes (eg. RHEL8). Try running this as root on the client:

update-crypto-policies --set DEFAULT:SHA1; /sbin/init 6

Also enure you get a ‘yes’ response from this comment:

sshd -T|egrep “pubkeyauthentication|pubkeyacceptedkeytypes”

pubkeyauthentication yes

And that ‘ssh-rsa’ is included in the output from from this command:

sshd -T |grep ssh-rsa

casignaturealgorithms …,…,…,ssh-rsa
hostbasedacceptedalgorithms ...,...,...ssh-rsa-cert-v01@openssh.com,…,…,…
hostkeyalgorithms ...,...,...ssh-rsa,ssh-rsa-cert-v01@openssh.com,…,…,…
pubkeyacceptedalgorithms ...,...,...ssh-rsa,ssh-rsa-cert-v01@openssh.com,…,…,…

If not, you may need to tweak /etc/crypto-policies/back-ends/opensshserver.config and / or /etc/ssh/sshd_config to add these algorithms.

NOTE: re-enabling SHA1 may violate your security policy.

14

As pointed out in Remote execution Authentication failure an up-to-date foreman server with remote execution won’t have any issues and doesn’t need any modification of the crypto policies on your el9 clients.

Only old foreman servers with old remote execution plugins using the netssh module instead of the standard ssh clients want to use sha1…