Remote execution fails (Permission denied)

Support
1

Problem:
Remote Execution fails with an error like follows

1:
sudo: unable to execute /var/tmp/foreman-ssh-cmd-c4fea95a-503c-489a-8fc7-84bb638c11db/script: Permission denied
2:
Exit status: 1

Expected outcome:

Script executes against remote host

Foreman and Proxy versions:
3.2

Foreman and Proxy plugin versions:
{Latest}

Distribution and version:
AlmaLinux 8.5

Other relevant data:

ssh user is set to “svcforemanrex”
Account has NOPASSWD set on remote hosts as well as the Foreman-Proxy’s id_rsa public key imported into authorized_keys
Effective user: root
Effective method: sudo

Am I missing something?

2

Check the logs on the remote host to see what goes wrong: either the ssh connection isn’t working or the sudo after that.

Is this a single server foreman installation or do you use different smart proxies for the ssh connections? The proxy basically does something like this as ssh connection

# sudo -u foreman-proxy ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy <ssh_user>@remote.example.com ‘sudo -u root /bin/ls’
3

Is your /var/tmp on the remote host mounted with noexec by any chance?

4

Yep, per security guidelines on both the foreman proxy and remote host,

/dev/mapper/almalinux-var_tmp /var/tmp ext4 defaults,nosuid,noexec,nodev 1 2

Is there a way to specify an alternate path?

5

found the file I need to tweak,

/etc/foreman-proxy/settings.d/remote_execution_ssh.yml

adjusted path to /var/foreman/cache , which I created on the remote host and gave permissions 777
I’m now trying to figure out how I can adjust those permissions down a bit as remote_exec doesn’t seem to sudo when it creates the file.

6

Yes, that’s the file. However bear in mind that file is managed by the installer and your customizations will be discarded on subsequent installer runs. If you want to make this change persistent, pass --foreman-proxy-plugin-remote-execution-script-remote-working-dir /var/foreman/cache to the installer.

7

According to installation documentation at → Installing Foreman 3.2 server with Katello 4.4 plugin on Enterprise Linux the partition to give exec is /tmp, not /var/tmp. So has it moved from /tmp to /var/tmp?
What is the reason for using these directories anyway for scripts? It does not sound like best security practice. I mean the REX user has a home directory that I assume works to execute scripts in?

8

Looking at --foreman-proxy-plugin-remote-execution-ssh-remote-working-dir
If I want to set that to the home directory of the REX user, I just run:

foreman-installer --foreman-proxy-plugin-remote-execution-ssh-remote-working-dir <homedir>

on both external proxies and foreman?
Bad idea to use the REX user homedir?

9

“This is a requirement for the puppetserver service to work.”

10

No, it has been /var/tmp since like 2016 or so.

Well, the files put there are temporary, but might be theoretically infinite in size, tipping the scales in /var/tmp's favor over /tmp.

Then change it.

On every machine that runs a smart proxy with ssh feature. Depending on your setup, that might be all the machines.

Possibly. The working directory is set per-proxy, but the REX user can be changed per job. You may end up trying to put scripts into other user’s home directory.

11

I am creating my REX user as a system user with home directory in /opt/fmrex.
So as long as the REX user is the owner of this directory and /opt partition is mounted with noexec It feels pretty safe and dynamic. Even if I need to change the REX user for a host down the line, I just have to make sure that the other REX user owns the /opt/fmrex directory.

12

So ran this in my proxies

foreman-installer --foreman-proxy-plugin-remote-execution-ssh-remote-working-dir /opt/fmrex

and can confirm it works just like expected. Scripts created and ran by REX in /opt/fmrex now. Sorta nice to not have to mess with modding the mount permissions for /var/tmp on all these machines with NIST security policies in place that will complain if /var/tmp or /tmp do not have noexec.

Have also beef up security even more by removing NOPASSWD in sudo conf and added a password for REX user to elevate to sudo. That way if someone hacked the proxies, all they get is a basic user access using the ssh key.

Also added these to /etc/ssh/sshd_config to slim down access even further:

#Deny access for user fmrex from non Foreman proxy IP.
Match User fmrex Address *,!<Proxy IP>
    ForceCommand /bin/false

#Only allow user fmrex to log in with SSH key.
Match User fmrex
    PasswordAuthentication no