Remote job execution through ssh failed from foreman GUI

Support
5

Yes initially I had copy keys for root from foreman server but I have corrected that and copied foreman-proxy keys by this command:
ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub root@target.example.com
And I am able to connect using the command you shared but still getting the same error.
Note: I am using the internal smart proxy server so my foreman and smart proxy servers are same.
Secondly, I am using the default ‘Run command - script’ job template and running a simple command like ‘lscpu’. Effective user as ‘root’ and no other advanced fields.

6

OK, looks good so far.
Thinks I can think of that could be wrong here:

  • Does ~foreman-proxy/.ssh/ and it’s content (the SSH key files) belong to the foreman-proxy user? Permissions should look something like this:
# ls -l ~foreman-proxy/.ssh/
total 12
-rw-------. 1 foreman-proxy foreman-proxy   77 Dec  2  2016 config
-rw-------. 1 foreman-proxy foreman-proxy 1675 Jul 28  2016 id_rsa_foreman_proxy
-rw-r--r--. 1 foreman-proxy foreman-proxy  422 Jul 28  2016 id_rsa_foreman_proxy.pub
  • If the permissions are correct, check /var/log/foreman-proxy/proxy.log on the Foreman server as well as the relevant logs on the target system (like /var/log/secure, /var/log/audit or wherever your systems log login stuff) for any errors or messages relating to the SSH connections
7

The permissions look good both sides. Sorry but may be a stupid mistake, my server doesn’t have the config file so have i missed some configuration.
There are no logs related to the connection on target server so seems the connection isn’t hitting it.

8

I am a little bit confused. You have mentioned ~foreman-proxy/.ssh/ but isn’t the default foreman-proxy/ssh/?

Checking /etc/foreman-proxy/settings.d/remote_execution_ssh.yml on my foreman server:

# cat /etc/foreman-proxy/settings.d/remote_execution_ssh.yml 
---
:enabled: https
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
:local_working_dir: /var/tmp
:remote_working_dir: /var/tmp
:kerberos_auth: false

:cockpit_integration: true

# Whether to run remote execution jobs asynchronously
:mode: ssh

Thus, the manual test command from shell on the main server would be:

[root@foreman8 ~]# sudo -u foreman-proxy ssh -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy root@test.example.com date
Thu Jan  5 04:32:24 PM CET 2023

There should be something in /var/log/secure on the remote side for that.

9

~foreman-proxy/.ssh/ is a softlink of foreman-proxy/ssh/ so shouldn’t matter I guess.

[root@foreman-master ~]# ls -ld ~foreman-proxy/.ssh
lrwxrwxrwx. 1 root root 26 Nov 15 15:42 /usr/share/foreman-proxy/.ssh → /var/lib/foreman-proxy/ssh

10

I am able to resolve the authentication problem as I think DNS was not resolving the host name so I enabled ‘connect by IP’ parameter. Now the new error looks like this:

1: Error initializing command: RuntimeError - Failed to execute script on remote machine, exit code: 126.
2: Exit status: EXCEPTION
3: StandardError: Job execution failed

Also I am trying to look for some documentation related to one of the question you asked “Have you configured the remote execution smart-proxy for the hosts (either globally or on the subnets)?”
I need to know what configuration I need to do for that.

11

So manual SSH from the proxy/Foreman works and you can also elevate that shell to root?
Can you execute scripts manually from that shell?
Check the logs on the host, do you see successful log in from Foreman when you run the REX command?

12

Is by any chance you /var/tmp mounted with noexec?

1 Like
13

Yes manually I can ssh and execute scripts.
[root@foreman-master ~]# ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@10.1.235.54
Last login: Fri Jan 6 16:21:41 2023 from 10.1.235.60
[root@foreman-server ~]# vi test.sh
[root@foreman-server ~]# ./test.sh
-bash: ./test.sh: Permission denied
[root@foreman-server ~]# ls -ld ./test.sh
-rw-------. 1 root root 24 Jan 6 17:12 ./test.sh
[root@foreman-server ~]# chmod u+x test.sh
[root@foreman-server ~]# ./test.sh
Hello
[root@foreman-server ~]# umask
0077
/var/log/secure logs from target which suggest connection is successful when running invoking the remote job.

Jan 6 17:18:48 foreman-server sshd[417299]: Connection from 10.1.235.60 port 53932 on 10.1.235.54 port 22
Jan 6 17:18:48 foreman-server sshd[417299]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan 6 17:18:48 foreman-server sshd[417299]: Postponed publickey for root from 10.1.235.60 port 53932 ssh2 [preauth]
Jan 6 17:18:48 foreman-server sshd[417299]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan 6 17:18:48 foreman-server sshd[417299]: Accepted publickey for root from 10.1.235.60 port 53932 ssh2: RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0
Jan 6 17:18:48 foreman-server sshd[417299]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jan 6 17:18:48 foreman-server sshd[417299]: User child is on pid 417303
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Starting session: command for root from 10.1.235.60 port 53932 id 0
Jan 6 17:18:48 foreman-server sshd[417303]: Close session: user root from 10.1.235.60 port 53932 id 0

14

Seems ok to me.
[root@foreman-master ~]# mount|grep /tmp
/dev/mapper/rootvg-tmp on /tmp type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
[root@foreman-master ~]# grep -i /tmp /etc/fstab
/dev/mapper/rootvg-tmp /tmp xfs defaults,nodev,nosuid,exec 0 0

15

fapolicyd enabled on the host by any chance?

17

No it’s not enabled.

[root@foreman-master ~]# systemctl status fapolicyd
Unit fapolicyd.service could not be found.
[root@foreman-master ~]#
18

Was more thinking on the host, I assume “foreman-master” is the Foreman server?

19

Sorry forgot to mention I did check on both the servers. It’s disabled.

20

It seems to get a little bit confusing here. Let’s trace that systematically:

You are only having a single foreman server and no other external foreman proxies? (You have only a single smart proxy aka the main server listed in Infrastructure - Smart Proxies)

You can run ssh commands from the foreman server to your client as foreman-proxy:

[root@foreman8 ~]# sudo -u foreman-proxy ssh -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy root@test.example.com date
Thu Jan  5 04:32:24 PM CET 2023

As far as I can see, you have never tested as foreman-proxy user but only as root. Please also use the hostname not the IP address. Collect the log in /var/log/secure on the client for this run.

The run that via remote execution. As you getting an error, check the task status for the job, e.g. click on the host name on the job invocation page and then press the Task Details button. Post the output on the Error tab and the raw input on the Raw tab. Collect the log in /var/log/secure on the client for this attempt.

21

Yes you are correct I have a single foreman server(with internal smart proxy as you mentioned) with no external proxies.

[root@foreman-master ~]# sudo -u foreman-proxy ssh -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy root@foreman-server.com date
Mon Jan  9 14:40:26 CET 2023
[root@foreman-master ~]#

Output from the Error tab:

Output:

{}

Exception:

StandardError: Job execution failed

Backtrace:

/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matchers/abstract.rb:74:in `assigns'
/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:56:in `match_value'
/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:36:in `block in match?'
/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:35:in `each'
/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:35:in `match?'
/usr/share/gems/gems/algebrick-0.7.5/lib/algebrick/matching.rb:23:in `match'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/error.rb:13:in `new'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:512:in `set_error'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:475:in `error!'
/usr/share/gems/gems/foreman_remote_execution-8.1.2/app/lib/actions/remote_execution/run_host_job.rb:109:in `check_exit_status'
/usr/share/gems/gems/foreman_remote_execution-8.1.2/app/lib/actions/remote_execution/run_host_job.rb:79:in `block in finalize'
/usr/share/gems/gems/foreman_remote_execution-8.1.2/app/lib/actions/remote_execution/template_invocation_progress_logging.rb:20:in `block in with_template_invocation_error_logging'
/usr/share/gems/gems/foreman_remote_execution-8.1.2/app/lib/actions/remote_execution/template_invocation_progress_logging.rb:20:in `catch'
/usr/share/gems/gems/foreman_remote_execution-8.1.2/app/lib/actions/remote_execution/template_invocation_progress_logging.rb:20:in `with_template_invocation_error_logging'
/usr/share/gems/gems/foreman_remote_execution-8.1.2/app/lib/actions/remote_execution/run_host_job.rb:77:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:604:in `block (2 levels) in execute_finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:40:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:20:in `block in finalize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:23:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/load_setting_values.rb:25:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:19:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:19:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:25:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:25:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:603:in `block in execute_finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:602:in `execute_finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:78:in `run_step'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:63:in `dispatch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:70:in `block in run_in_sequence'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:70:in `all?'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:70:in `run_in_sequence'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:59:in `dispatch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:28:in `block in finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:29:in `block in finalize_phase'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:29:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:27:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:143:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Output from the Raw tab:

{
  "host": {
    "id": 2,
    "name": "foreman-server.local"
  },
  "job_category": "Commands",
  "description": "Run date",
  "job_invocation_id": 42,
  "job_features": [],
  "proxy_id": 1,
  "delegated_action_id": 2,
  "with_event_logging": true,
  "current_request_id": "63903f48-5907-4b65-9b55-41e4c21abaa6",
  "current_timezone": "Europe/Paris",
  "current_user_id": 4,
  "current_organization_id": 1,
  "current_location_id": 2
}
22

/var/log/secure log from client:

Jan  9 14:40:26 foreman-server sshd[456874]: Connection from 10.1.235.60 port 60868 on 10.1.235.54 port 22
Jan  9 14:40:26 foreman-server sshd[456874]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan  9 14:40:26 foreman-server sshd[456874]: Postponed publickey for root from 10.1.235.60 port 60868 ssh2 [preauth]
Jan  9 14:40:26 foreman-server sshd[456874]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan  9 14:40:26 foreman-server sshd[456874]: Accepted publickey for root from 10.1.235.60 port 60868 ssh2: RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0
Jan  9 14:40:26 foreman-server sshd[456874]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jan  9 14:40:26 foreman-server sshd[456874]: User child is on pid 456878
Jan  9 14:40:26 foreman-server sshd[456878]: Starting session: command for root from 10.1.235.60 port 60868 id 0
Jan  9 14:40:26 foreman-server sshd[456878]: Close session: user root from 10.1.235.60 port 60868 id 0
Jan  9 14:40:26 foreman-server sshd[456878]: Received disconnect from 10.1.235.60 port 60868:11: disconnected by user
Jan  9 14:40:26 foreman-server sshd[456878]: Disconnected from user root 10.1.235.60 port 60868
Jan  9 14:40:26 foreman-server sshd[456874]: pam_unix(sshd:session): session closed for user root
23

Is there anything logged in /var/log/secure for the remote execution?

Why is the server name in foreman foreman-server.local? You have used foreman-server.com before. Looks like a incomplete hostname or some incorrect set hostname.

24

/var/log/secure for remote execution job:

Jan  9 14:44:03 foreman-server sshd[456933]: Connection from 10.1.235.60 port 60882 on 10.1.235.54 port 22
Jan  9 14:44:03 foreman-server sshd[456933]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan  9 14:44:03 foreman-server sshd[456933]: Postponed publickey for root from 10.1.235.60 port 60882 ssh2 [preauth]
Jan  9 14:44:03 foreman-server sshd[456933]: Accepted key RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYhBii6ulr+0 found at /root/.ssh/authorized_keys:1
Jan  9 14:44:03 foreman-server sshd[456933]: Accepted publickey for root from 10.1.235.60 port 60882 ssh2: RSA SHA256:mqVHwwV0mMk/lWJPeV/2po1kUUQAkxYWYh
Bii6ulr+0
Jan  9 14:44:03 foreman-server sshd[456933]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jan  9 14:44:03 foreman-server sshd[456933]: User child is on pid 456937
Jan  9 14:44:03 foreman-server sshd[456937]: Starting session: command for root from 10.1.235.60 port 60882 id 0
Jan  9 14:44:03 foreman-server sshd[456937]: Close session: user root from 10.1.235.60 port 60882 id 0
Jan  9 14:44:03 foreman-server sshd[456937]: Starting session: command for root from 10.1.235.60 port 60882 id 0
Jan  9 14:44:03 foreman-server sshd[456937]: Close session: user root from 10.1.235.60 port 60882 id 0
Jan  9 14:44:03 foreman-server sshd[456937]: Starting session: command for root from 10.1.235.60 port 60882 id 0
Jan  9 14:44:03 foreman-server sshd[456937]: Close session: user root from 10.1.235.60 port 60882 id 0
Jan  9 14:44:03 foreman-server sshd[456937]: Starting session: command for root from 10.1.235.60 port 60882 id 0
Jan  9 14:44:03 foreman-server sshd[456937]: Close session: user root from 10.1.235.60 port 60882 id 0

Sorry I am not using DNS and there was a wrong entry in /etc/hosts file. I have corrected that but error is same.

[root@foreman-master ~]# sudo -u foreman-proxy ssh -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy root@foreman-server.local date
Mon Jan  9 15:09:16 CET 2023
[root@foreman-master ~]#
25

I got the issue resolved. /var/tmp was mounted with noexec option for target. I knew on foreman server we have to use exec but was not sure if it has to be for the targets as well. aruzicka mentioned it earlier but I only checked on foreman server. Remote job execution through ssh failed from foreman GUI - #12 by aruzicka
Thank you guys for all your quick responses but it was a miss from my end and a new learning as I am just getting started with this. I think I will se you guys around more often.
Solution:

1. My first error was ` **Could not establish connection to remote host**`. I came to know that since I was not using DNS service the client fqdn was not getting resolved so job was failing. So either entry in /etc/hosts or enabling the `connect by IP` option in `Administer -> Settings -> Remote Execution` tab worked.
2. The second error `Failed to execute script on remote machine` got resolved by changing the mount option `noexec` to `exec` for `/var/tmp` and ` /tmp` on client.