Remote Execution Fails

Problem: Executing Commands on remote Host Using " [Run Command - SSH Default - SSH]" it fails every time

Expected outcome: It Should be executed successfully

Foreman and Proxy versions: Forman Version 3.1.2

Foreman and Proxy plugin versions: Foreman Remote Execution Version 5.0.6

Distribution and version: CentOS 7

Other relevant data:

Just “it fails” is not enough. You will need to provide more details - job output and logs at least

Hi Aruzicka,

Thanks For quick response.

################### LOGS #######################################

Host,stdout,stderr,debug,Result,Finished
xdevsl.admiral.es,Exit status: EXCEPTION,"",“Error initializing command: RuntimeError - Unable to create directory on remote system /var/tmp/foreman-ssh-cmd-1e08e68a-413c-4b80-994b-0639004e6dda: exit code: 255
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket “”/var/tmp/foreman-proxy/foreman-ssh-cmd-1e08e68a-413c-4b80-994b-0639004e6dda/socket”" does not exist
debug1: Connecting to xdevsl.admiral.es [10.93.0.55] port 22.
debug1: Connection established.
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy type 1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to xdevsl.admiral.es:22 as ‘platform’
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:pSxtpUT54jjoojyqDMJwWhRC/XOKjoILxqxFiBI1ca0
debug1: Host ‘xdevsl.admiral.es’ is known and matches the ECDSA host key.
debug1: Found key in /usr/share/foreman-proxy/.ssh/known_hosts:28
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received


( ____ |\ /|__ /( ____ ( ____
| ( /| ) ( | ) ( | ( /| ( /
| (
| | | | | | | | | (_____
| ) | | | | | | | | ____ (___ )
| ( | | | | | | | | _ ) ) |
| (/| () |) (| () |/_) |
(
/()_/()_____)


  • WARNING !!! *

  • This system is for the use of authorized users only. *

  •                                                                        *
    
  • Individuals using this network without authority, or in excess of their *

  • authority, are subject to legal action. *

  •                                                                        *
    

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
",error,2022-8-2 6:05:24 +0000

If I’m reading it right, then the proxy tried to perform a publickey based authentication and that failed. Are ssh keys set up correctly for the user you’re trying to connect as?

Could you please let me know how to check that because I am setting up this for first time.
Earlier we had a foreman of version 1.17 but I installed a newer version and trying to replicate all things exactly same as older one.

On your foreman server (or proxy if applicable) verify the ssh key is working:

[root@foreman ~]# sudo -u foreman-proxy ssh root@c7build.example.com -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -v

That’s basically what’s the plugin is running. If that fails or asks for a password, it’s not working. So far, it seems as if the public key in /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub has not been added to authorized_keys for root on the remote client…

The above command is not working but it is also not working on the older formean where remote execution works fine. I am attaching the screenshot of the older foreman remote execution task/template.

Hello!
As @gvde has stated - can you verify and confirm that the public key:
/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub exists in /root/.ssh/authorized_keys of the target host?

It doesn’t exist. But earlier as well in older version of foreman it doesn’t exist in any of the target hosts but its working fine.

So that’s why I am unable to find out any specific issue.

Not sure how you performed the upgrade, but I guess the priv/pub key pairs got regenerated, so they are different now. You definitely have to put the actual pub key to the authorized_keys file of your target host in order to be able to successfully authenticate.

I didn’t upgrade the older version instead I created a new host with 3.1.2 version. So if you want anything from older version host I could provide the info.

And thanks for the all the help till now. Actually I am trying to solve this issue from last 2 weeks but couldn’t find anything that’s why I reached to you guys for the help.

You forget to mention that you don’t ssh into user “root” but user “platform” instead and use sudo after that. So the command which must work is:

[root@foreman ~]# sudo -u foreman-proxy ssh root@c7build.example.com -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -v -l platform

The relevant authorized_keys file should be in ~platform/.ssh/authorized_keys on the target hosts.

If it is not working, please post the full output of the above ssh command. It’s not very helpful if you only write “is not working”…

Still not working .Please find the output of above command below from New Foreman As well old one.

####################### New Foreman #########################################
[root@xproorchestrator05 ~]# sudo -u foreman-proxy ssh root@c7build.example.com -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -v -l platform
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
ssh: Could not resolve hostname c7build.example.com: Name or service not known
[root@xproorchestrator05 ~]# sudo -u foreman-proxy ssh root@xdevsl.admiral.es -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -v -l platform
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to xdevsl.admiral.es [10.93.0.55] port 22.
debug1: Connection established.
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy type 1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to xdevsl.admiral.es:22 as ‘platform’
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:pSxtpUT54jjoojyqDMJwWhRC/XOKjoILxqxFiBI1ca0
debug1: Host ‘xdevsl.admiral.es’ is known and matches the ECDSA host key.
debug1: Found key in /usr/share/foreman-proxy/.ssh/known_hosts:28
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received


( ____ |\ /|__ /( ____ ( ____
| ( /| ) ( | ) ( | ( /| ( /
| (
| | | | | | | | | (_____
| ) | | | | | | | | ____ (___ )
| ( | | | | | | | | _ ) ) |
| (/| () |) (| () |/_) |
(
/()_/()_____)


  • WARNING !!! *

  • This system is for the use of authorized users only. *

  •                                                                        *
    
  • Individuals using this network without authority, or in excess of their *

  • authority, are subject to legal action. *

  •                                                                        *
    

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:662)

debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:662)

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: password
platform@xdevsl.admiral.es’s password:

###########################################################################

####################### OLD FOREMAN #####################################

[root@xproorchestrator03:~#] sudo -u foreman-proxy ssh root@xdevsl.admiral.es -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -v -l platform
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to xdevsl.admiral.es [10.93.0.55] port 22.
debug1: Connection established.
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy type 1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to xdevsl.admiral.es:22 as ‘platform’
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:pSxtpUT54jjoojyqDMJwWhRC/XOKjoILxqxFiBI1ca0
debug1: Host ‘xdevsl.admiral.es’ is known and matches the ECDSA host key.
debug1: Found key in /usr/share/foreman-proxy/.ssh/known_hosts:608
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received


( ____ |\ /|__ /( ____ ( ____
| ( /| ) ( | ) ( | ( /| ( /
| (
| | | | | | | | | (_____
| ) | | | | | | | | ____ (___ )
| ( | | | | | | | | _ ) ) |
| (/| () |) (| () |/_) |
(
/()_/()_____)


  • WARNING !!! *

  • This system is for the use of authorized users only. *

  •                                                                        *
    
  • Individuals using this network without authority, or in excess of their *

  • authority, are subject to legal action. *

  •                                                                        *
    

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:662)

debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:662)

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: password
platform@xdevsl.admiral.es’s password:

#############################################################################

And the public key is in ~platform/.ssh/authorized_keys on the target host?

Hey its not possible to add the public keys in target hosts as there are almost more than 1000 hosts and it is working in old foreman without adding the keys.

So could there be any other way without adding the public keys.

Please simply answer the question: And the public key is in ~platform/.ssh/authorized_keys on the target host?

It’s really difficult to help you, if you don’t answer the questions or post the output of commands you are supposed to run. This information is vital to find out what’s exactly wrong. Anything else is guessing and that doesn’t really help and is often a waste of time…

So again: is the public key in the authorized_keys file? Is there a authorized_keys file at all for that user on the target hosts. If there is, verify that it contains the public key from the old foreman server.

And as rplevka before pointed out: a new installation automatically creates new ssh keys…

No there is no public key present from either of the foreman server on any of the target hosts.

And if you guys know how to change the default location of keys I mean instead of picking the key from /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy I could mention any other key because platform user keys are present on all servers.

If that’s true, then you have to find out how the old server gets a working ssh connection into the target hosts. By default, it would use an SSH key generated on the foreman server. You would have to redistribute the SSH public key to the target host into the authorized_keys file of the target user account. To that extent, that’s basic SSH public key authentication and nothing else.

If your servers and targets do that differently then you have to find out how. That’s not a simple standard setup anymore and everyone guessing here what you might have is futile.

So basically, there is only a generic approach to find out what happens on the old server to figure out how to get it working on the new. You have to figure out where the old foreman server stores its ssh keys. That should be in the logs, maybe debug logs are required. The ssh verbose log tells you where it picks up the keys.

Once found, you can verify it’s the correct keys by using the above ssh command on your old foreman server pointing to the ssh keys you have identified. Only if that command works, it’s the right place. If not, it’s the wrong key and you have to keep digging.

If followed the docs for the remote execution plugin the place to look for would be ~foreman-proxy/.ssh/. But you have to find out where it is on your old foreman server.

Of course, there may be different places if you have a more complex setup, e.g. ipa/ldap/sssd storing ssh keys, etc. But you should know about that.

So again: first thing to do is to get a working ssh call running on the old foreman server to figure out how it works. Once you know that it should be easy to find out how to get it working on the new server.

And beyond that: before you wrote:

That sounds to me as if you don’t have any keys in authorized_keys of user platform on the target hosts.

Then you wrote:

That sounds to me as if there are keys present in authorized_keys for user platform on the target hosts. But how do you know it’s not a key added for the old foreman server or how do you know none of those keys is used by your old foreman server.

So what you write sounds all quite confusing and if you don’t know what users from which hosts can ssh into what targets using ssh keys…

Below is the output when I changed the key of platform user instead of default one and it worked fine. So I just want to know is there any way we could point the foreman to use our desired key instead of the default one.

#############################################################################
[root@xproorchestrator05 ssh]# sudo -u foreman-proxy ssh root@xdevsl.admiral.es -i /var/lib/foreman-proxy/ssh/id_rsa -v -l platform
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to xdevsl.admiral.es [10.93.0.55] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/foreman-proxy/ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to xdevsl.admiral.es:22 as ‘platform’
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:pSxtpUT54jjoojyqDMJwWhRC/XOKjoILxqxFiBI1ca0
debug1: Host ‘xdevsl.admiral.es’ is known and matches the ECDSA host key.
debug1: Found key in /usr/share/foreman-proxy/.ssh/known_hosts:28
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received


( ____ |\ /|__ /( ____ ( ____
| ( /| ) ( | ) ( | ( /| ( /
| (
| | | | | | | | | (_____
| ) | | | | | | | | ____ (___ )
| ( | | | | | | | | _ ) ) |
| (/| () |) (| () |/_) |
(
/()_/()_____)


  • WARNING !!! *

  • This system is for the use of authorized users only. *

  •                                                                        *
    
  • Individuals using this network without authority, or in excess of their *

  • authority, are subject to legal action. *

  •                                                                        *
    

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:662)

debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:662)

debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/foreman-proxy/ssh/id_rsa
debug1: Authentication succeeded (publickey).
Authenticated to xdevsl.admiral.es ([10.93.0.55]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Tue Aug 2 14:23:42 2022 from 192.168.240.85


( ____ |\ /|__ /( ____ ( ____
| ( /| ) ( | ) ( | ( /| ( /
| (
| | | | | | | | | (_____
| ) | | | | | | | | ____ (___ )
| ( | | | | | | | | _ ) ) |
| (/| () |) (| () |/_) |
(
/()_/()_____)


  • WARNING !!! *

  • This system is for the use of authorized users only. *

  •                                                                        *
    
  • Individuals using this network without authority, or in excess of their *

  • authority, are subject to legal action. *

  •                                                                        *
    

[platform@xdevsl:~$] exitdebug1: channel 0: free: client-session, nchannels 1
Connection to xdevsl.admiral.es closed by remote host.
Connection to xdevsl.admiral.es closed.
Transferred: sent 2876, received 4772 bytes, in 314.6 seconds
Bytes per second: sent 9.1, received 15.2
debug1: Exit status -1
[root@xproorchestrator05 ssh]#