Remote Execution fails immediately with " Could not establish connection"

Support
1

Hi Everyone,

I’m trying to install and configure the new Foreman installation, as we are migrating to Debian 11.
Unfortunately, I can’t make Remote Execution working.

Problem:

image
image1221×120 15.3 KB

Foreman and Proxy versions:
Foreman - 3.7.0
image

Distribution and version:
Debian GNU/Linux 11 (bullseye)

Other relevant data:
When I’m trying to run any command via the WebUI it immediately fails.

When I’m running command:

sudo -u foreman-proxy ssh root@chd-prd-foreman02 -i /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy -v

there is no problem with connecting to another host:

root@chc-prd-vforeman01:~# sudo -u foreman-proxy ssh root@chd-prd-foreman02 -i /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy -v
OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n  15 Mar 2022
debug1: Reading configuration data /usr/share/foreman-proxy/.ssh/config
debug1: /usr/share/foreman-proxy/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to chd-prd-foreman02 [10.9.64.196] port 22.
debug1: Connection established.
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 0
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 0
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to chd-prd-foreman02:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:tCVhvHfeNaDmTm2EfdiXpPQFuJNf+0dYmCS9U8ShkDM
Warning: Permanently added 'chd-prd-foreman02,10.9.64.196' (ECDSA) to the list of known hosts.
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:9pKxK6rR53AVprsqCWmUk8CIa6hrpyVx4/qxvbYPEbg explicit
debug1: Will attempt key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:9pKxK6rR53AVprsqCWmUk8CIa6hrpyVx4/qxvbYPEbg explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_996)


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_996)


debug1: Next authentication method: publickey
debug1: Offering public key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:9pKxK6rR53AVprsqCWmUk8CIa6hrpyVx4/qxvbYPEbg explicit
debug1: Server accepts key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:9pKxK6rR53AVprsqCWmUk8CIa6hrpyVx4/qxvbYPEbg explicit
debug1: Authentication succeeded (publickey).
Authenticated to chd-prd-foreman02 ([10.9.64.196]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Tue Oct 31 06:09:57 2023 from chc-prd-vforeman01.example.com
[root@chd-prd-foreman02 ~]#

This is my remote_execution_ssh.yml config:

root@chc-prd-vforeman01:~# cat /etc/foreman-proxy/settings.d/remote_execution_ssh.yml 
---
:enabled: https
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
:local_working_dir: /var/tmp
:remote_working_dir: /var/tmp
:kerberos_auth: false

:cockpit_integration: true

# Whether to run remote execution jobs asynchronously
:mode: ssh

and there is a symlink to /usr/share/foreman-proxy/ssh:

root@chc-prd-vforeman01:~# ls -lah /usr/share/foreman-proxy/ | grep ssh
lrwxrwxrwx   1 foreman-proxy foreman-proxy   26 Oct 16 04:08 .ssh -> /var/lib/foreman-proxy/ssh

I will appreciate any advises.

2

Hi,
if you bump log_level in /etc/foreman-proxy/settings.yml to DEBUG, restart foreman-proxy service and run the job again, /var/log/foreman-proxy/proxy.log should contain the exact ssh commands the proxy is trying to execute so you should be able to follow along.

1 Like
3

Thank you. I think this is helpful advice, but I’m not sure how to solve this yet:

2023-10-31T09:28:55  [D] Running: ssh -o User=root -o Port=22 -o IdentityFile=/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=error -o ControlMaster=auto -o ControlPath=/var/tmp/bdaf7b7f-370e-4603-8bd0-4fba4bc8afb9 -o ControlPersist=yes -o ProxyCommand=none -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -o PreferredAuthentications=publickey -o NumberOfPasswordPrompts=0 chd-prd-foreman02.example.com true
2023-10-31T09:28:55  [D] unix_listener: cannot bind to path /var/tmp/bdaf7b7f-370e-4603-8bd0-4fba4bc8afb9.tm0GPw9bgQFTxgbf: Permission denied
2023-10-31T09:28:55  [D] Failed to establish connection using authentication method publickey

We are using remote execution with the same parameters in our old Foreman implementation and we don’t see issues like that.

4 
unix_listener: cannot bind to path /var/tmp/bdaf7b7f-370e-4603-8bd0-4fba4bc8afb9.tm0GPw9bgQFTxgbf: Permission denied

This looks suspicious, do you have any non-standard mount options on /var/tmp or /tmp?

1 Like
5

Yes, it was the issue.
It seems like I had just default Debian 11 permissions for /var/tmp.

Original:

drwxr-xr-x 11 root root  4.0K Oct 31 09:30 tmp

So, I just run

chmod 777 /var/tmp

and it works without any issues now.
Thank you for your help @aruzicka