SSH remote operations only work if I input the root user (and its password)

I can run remote Ansible jobs without a password and can su to foreman and then SSH to root@client using the key, but the proxy refuses to use it.

I expect to be able to run an SSH remote job using my key

I am running Foreman 2.0 and have these proxy RPMs
tfm-rubygem-smart_proxy_openscap-0.7.2-3.fm2_0.el7.noarch
foreman-proxy-2.0.0-1.el7.noarch
gssproxy-0.7.0-28.el7.x86_64
tfm-rubygem-smart_proxy_remote_execution_ssh-0.2.1-4.fm2_0.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-3.fm2_0.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.0.0-2.fm2_0.el7.noarch
sssd-proxy-1.16.4-37.el7.x86_64
tfm-rubygem-smart_proxy_dynflow_core-0.2.4-2.fm2_0.el7.noarch
foreman.domain.local-foreman-proxy-client-1.0-1.noarch
tfm-rubygem-smart_proxy_dynflow-0.2.4-3.fm2_0.el7.noarch
foreman.domain.local-foreman-proxy-1.0-1.noarch
libproxy-0.4.11-11.el7.x86_64

I am running Centos 7.8 on a libvirt VM.

Originally the server was a member of an IPA domain but I have since unistalled it and edited /etc/ssh/ssh_config to comment out:

ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

When execute a remote job. I see:
Error initializing command: Net::SSH::AuthenticationFailed - Authentication failed for user root@workloads.bluefinch.local

Hi,
where did you put the key and what are the permissions on it?

Hi, sorry for the delay in my reply.
I used the standard commands from the manual:

# sudo -u foreman-proxy ssh-keygen -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -N ''
# ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub <client>

My keys look like this:

[root@foreman .ssh]# ls -ld /usr/share/foreman-proxy/.ssh
lrwxrwxrwx. 1 root root 22 Apr 26 19:49 /usr/share/foreman-proxy/.ssh -> /var/foreman-proxy/ssh
[root@foreman .ssh]# ls -al /var/foreman-proxy/ssh
total 20
drwxr-x---. 2 foreman-proxy foreman-proxy  112 May 12 19:25 .
drwxr-xr-x. 7 root          root            85 Apr 26 19:49 ..
-rw-------. 1 foreman-proxy foreman-proxy  201 May  3 18:35 config
-rw-------. 1 foreman-proxy foreman-proxy 1679 Apr 28 10:39 id_rsa
-rw-------. 1 foreman-proxy foreman-proxy 1679 May 12 19:18 id_rsa_foreman_proxy
-rw-r--r--. 1 foreman-proxy foreman-proxy  419 May 12 19:18 id_rsa_foreman_proxy.pub
-rw-r--r--. 1 foreman-proxy foreman-proxy  410 Apr 28 10:39 id_rsa.pub

What is wierd is if I test it:

# sudo -u foreman-proxy -s /bin/bash
bash-4.2$ ssh root@sensu.domain.local date
Warning: Permanently added 'sensu.domain.local,192.168.0.118' (ECDSA) to the list of known hosts.
Wed May 13 12:32:25 CEST 2020
bash-4.2$

even though I have already done this test several times.

I think the ssh root@$host date test uses ~foreman-proxy/id_rsa as a private key. Could you check on the target host that it really has the contents of ~foreman-proxy/id_rsa_foreman_proxy.pub in /root/.ssh/authorized_keys?

Hi,

I have checked this on all my clients and they definitely have this key in the authorized_keys file.

Hello,

this reminds me Bug #25481: Foreman fails to execute ansible commands when connected to ipa server - Installer - Foreman however the error message seems different. Check your /etc/foreman-proxy/ansible.cfg, namely the ssh_args in the [ssh_connection]. If the target is debian based system, I had to add -C -o ControlMaster=auto -o ControlPersist=60s in there. OTOH you’re already running 2.0, where it should have been fixed.

No more ideas, sorry. Perhaps try reproducing the ssh connection under foreman-proxy user without running the new bash instance.

Hi Marek,

I did start by installing my Foreman server as an IPA client but unistalled the IPA software to try to fix this issue.

My ansible.cfg looks like this and I am running Centos 7 with the latest version of Foreman 2.x:

[defaults]
callback_whitelist = foreman
local_tmp = /tmp
host_key_checking = False
stdout_callback = yaml
roles_path = /etc/ansible/roles:/usr/share/ansible/roles

[callback_foreman]
url = https://foreman.domain.local
ssl_cert = /etc/foreman-proxy/foreman_ssl_cert.pem
ssl_key = /etc/foreman-proxy/foreman_ssl_key.pem
verify_certs = /etc/foreman-proxy/foreman_ssl_ca.pem

[ssh_connection]
ssh_args = -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s

How do you test foremen-proxy without Bash?

config looks reasonable to me

How do you test foremen-proxy without Bash?

what I mean was

instead of

# sudo -u foreman-proxy -s /bin/bash
bash-4.2$ ssh root@sensu.domain.local date

try just

sudo -u foreman-proxy ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@sensu.domain.local date

new new bash instance may behave differently, but just a wild guess…