**Problem:
Remote job execution through ssh failed from foreman GUI.
Starting to use foreman and have registered a client. Need to test remote execution job by simply running a command but the job fails. I have enabled the remote execution plugins and ssh plugin as well.
**
**Expected outcome:
Output of the command given during job scheduling
**
Foreman and Proxy versions:
3.5.1
**Foreman and Proxy plugin versions:
foreman-tasks
7.1.0
foreman_ansible
10.1.0
foreman_discovery
22.0.2
foreman_remote_execution
8.1.2
foreman_setup
8.0.1
katello
4.7.0
**
Distribution and version:
Installed on a virtual machine-RHEL8.7
Are you getting any error messages? Just saying âfailesâ is not very concrete.
If you mean the Job is reported as âfailedâ in Foreman GUI, you can click your registered hosts name in the jobâs overview (monitor â jobs â click your test job; or: The page you see directly after starting the job) to see the output. Not that jobs are marked as failed both when Foreman cannot execute the job due to some failure and if the job returns an exit code other than 0.
In case you see an error thatâs not inside your test script, check /var/log/foreman/production.log and /var/log/foreman-proxy/proxy.log for any related messages during the timeframe of the job.
Just some things that come to mind:
Have you configured the remote execution smart-proxy for the hosts (either globally or on the subnets)?
Have you distibuted the SSH Keys for the smart-proxy to the hosts?
If you have any firewalls running, are they configured to allow SSH connections?
You might want to take a look at the docs for some additional pointers, the documentation on the remote-execution plugin page seems to be quite outdated.
Hi, thanks for the reply. I did share the production.log in the original post but I think somehow it did not get posted. While I am reviewing the link you shared and some more docs let me share the error message - Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried publickey
** 2:** Exit status: EXCEPTION
** 3:** StandardError: Job execution failed
That does indeed look like you have not set up the SSH keys for authentication correctly.
On your Foreman server, you can run the following command and check if the SSH connection works: ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@<your test host>
Yes initially I had copy keys for root from foreman server but I have corrected that and copied foreman-proxy keys by this command:
ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub root@target.example.com
And I am able to connect using the command you shared but still getting the same error.
Note: I am using the internal smart proxy server so my foreman and smart proxy servers are same.
Secondly, I am using the default âRun command - scriptâ job template and running a simple command like âlscpuâ. Effective user as ârootâ and no other advanced fields.
If the permissions are correct, check /var/log/foreman-proxy/proxy.log on the Foreman server as well as the relevant logs on the target system (like /var/log/secure, /var/log/audit or wherever your systems log login stuff) for any errors or messages relating to the SSH connections
The permissions look good both sides. Sorry but may be a stupid mistake, my server doesnât have the config file so have i missed some configuration.
There are no logs related to the connection on target server so seems the connection isnât hitting it.
I am able to resolve the authentication problem as I think DNS was not resolving the host name so I enabled âconnect by IPâ parameter. Now the new error looks like this:
Also I am trying to look for some documentation related to one of the question you asked âHave you configured the remote execution smart-proxy for the hosts (either globally or on the subnets)?â
I need to know what configuration I need to do for that.
So manual SSH from the proxy/Foreman works and you can also elevate that shell to root?
Can you execute scripts manually from that shell?
Check the logs on the host, do you see successful log in from Foreman when you run the REX command?
Yes manually I can ssh and execute scripts.
[root@foreman-master ~]# ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@10.1.235.54
Last login: Fri Jan 6 16:21:41 2023 from 10.1.235.60
[root@foreman-server ~]# vi test.sh
[root@foreman-server ~]# ./test.sh
-bash: ./test.sh: Permission denied
[root@foreman-server ~]# ls -ld ./test.sh
-rw-------. 1 root root 24 Jan 6 17:12 ./test.sh
[root@foreman-server ~]# chmod u+x test.sh
[root@foreman-server ~]# ./test.sh
Hello
[root@foreman-server ~]# umask
0077
/var/log/secure logs from target which suggest connection is successful when running invoking the remote job.
Jan 6 17:18:48 foreman-server sshd[417299]: Connection from 10.1.235.60 port 53932 on 10.1.235.54 port 22
Jan 6 17:18:48 foreman-server sshd[417299]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan 6 17:18:48 foreman-server sshd[417299]: Postponed publickey for root from 10.1.235.60 port 53932 ssh2 [preauth]
Jan 6 17:18:48 foreman-server sshd[417299]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan 6 17:18:48 foreman-server sshd[417299]: Accepted publickey for root from 10.1.235.60 port 53932 ssh2: RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0
Jan 6 17:18:48 foreman-server sshd[417299]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jan 6 17:18:48 foreman-server sshd[417299]: User child is on pid 417303
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0
It seems to get a little bit confusing here. Letâs trace that systematically:
You are only having a single foreman server and no other external foreman proxies? (You have only a single smart proxy aka the main server listed in Infrastructure - Smart Proxies)
You can run ssh commands from the foreman server to your client as foreman-proxy:
[root@foreman8 ~]# sudo -u foreman-proxy ssh -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy root@test.example.com date
Thu Jan 5 04:32:24 PM CET 2023
As far as I can see, you have never tested as foreman-proxy user but only as root. Please also use the hostname not the IP address. Collect the log in /var/log/secure on the client for this run.
The run that via remote execution. As you getting an error, check the task status for the job, e.g. click on the host name on the job invocation page and then press the Task Details button. Post the output on the Error tab and the raw input on the Raw tab. Collect the log in /var/log/secure on the client for this attempt.
