Remote execution give me "Operation not permitted" from script-wrapper

[tedevil]

Running on Foreman 3.3/Katello 4.5.
Have started to run some tests on a hardened AlmaLinux 9 and for some reason I get this error from a remote execution run:

 1: sh: line 1: /opt/fmrex/foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2/script-wrapper: Operation not permitted
 2: Exit status: 126

The command sent was just “date” so I expected no problems.
If I log in manually from the proxy, I can log in with ssh using the ~foreman-proxy/.ssh/id_rsa_foreman_proxy key, use sudo to elevate myself to root and issue the command without problems so I am a little confused what is not allowed.

I fetched the script-wrapper file and it contains:

echo $$ > /opt/fmrex/foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2/pid; exec "$@";

Not totally sure what these commands are supposed to do but as root I have no problems executing them.
Logging however show the pid file was never created during remote execution so this command is my suspect. Anyone knows the details on how this is working. Was left with this evidence:

[root@sees-almatest foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2]# ls -la
total 20
drwxr-x---. 2 root  root  112 Jul 30 21:26 .
drwxr-x---. 3 fmrex fmrex  66 Jul 30 20:58 ..
-rw-r-----. 1 root  root   51 Jul 30 20:58 effective-user-test
-rw-r-----. 1 root  root    4 Jul 30 20:58 exit_code
-rw-r-----. 1 root  root    0 Jul 30 20:58 output
-rw-r-----. 1 root  root    5 Jul 30 20:58 script
-rw-r-----. 1 root  root   90 Jul 30 20:58 script-wrapper
-rw-r-----. 1 root  root   20 Jul 30 20:58 test
[root@sees-almatest foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2]# grep . *
effective-user-test:#!/bin/sh
effective-user-test:exec sudo -p 'rex login: ' -u root  true
exit_code:126
script:date
script-wrapper:echo $$ > /opt/fmrex/foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2/pid; exec "$@";
test:#!/bin/sh
test:exec true

[rplevka]

Just a hint, that the exit code 126 indicates that the file it is trying to execute is missing the executable flag - it’s also visible in the file listing you pasted.

[tedevil]

Did the same file pulling from a non hardeded AlmaLinux 8 and it shows they have indeed other permissions:

[fmrex@sees-fmtestvm2 foreman-ssh-cmd-6e497bc4-bab1-4188-8938-cfa5df039d1f]$ ls -la
total 28
drwxr-xr-x. 2 root root 123 Jul 30 21:24 .
drwxr-xr-x. 9 root root 265 Jul 30 21:24 ..
-r-xr-xr-x. 1 root root  51 Jul 30 21:24 effective-user-test
-rw-r--r--. 1 root root   2 Jul 30 21:24 exit_code
-rw-r--r--. 1 root root  29 Jul 30 21:24 output
-rw-r--r--. 1 root root   7 Jul 30 21:24 pid
-r-xr-xr-x. 1 root root   5 Jul 30 21:24 script
-r-xr-xr-x. 1 root root  90 Jul 30 21:24 script-wrapper
-r-xr-xr-x. 1 root root  20 Jul 30 21:24 test
[fmrex@sees-fmtestvm2 foreman-ssh-cmd-6e497bc4-bab1-4188-8938-cfa5df039d1f]$ grep . *
effective-user-test:#!/bin/sh
effective-user-test:exec sudo -p 'rex login: ' -u root  true
exit_code:0
output:Sat Jul 30 21:24:15 UTC 2022
pid:244690
script:date
script-wrapper:echo $$ > /opt/fmrex/foreman-ssh-cmd-6e497bc4-bab1-4188-8938-cfa5df039d1f/pid; exec "$@";
test:#!/bin/sh
test:exec true

How is remote executing setting the permissions on these files?

[rplevka]

Hi.
I believe it is uploading the files with permissions 555 here:

theforeman/smart_proxy_remote_execution_ssh/blob/master/lib/smart_proxy_remote_execution_ssh/runners/script_runner.rb#L183

Anyway, maybe the hardening mounts the destination with noexec flag?
If that is true, you might need to reconfigure REX to use different working path, just like described here:

Remote execution fails (Permission denied)

Yes, that’s the file. However bear in mind that file is managed by the installer and your customizations will be discarded on subsequent installer runs. If you want to make this change persistent, pass --foreman-proxy-plugin-remote-execution-script-remote-working-dir /var/foreman/cache to the installer.

Hope that helps!

[tedevil]

The partition has not the noexec flag (first thing I checked ).

Anyway, was digging around in the “OpenSCAP Evaluation Report” and came across an activation of fapolicyd “(File Access Policy Daemon) implements application whitelisting to decide file access rights”. Seen it was enabled. Stopped it and ran “fapolicyd --debug-deny”, ran the rex command again and got:

rule=13 dec=deny_audit perm=execute auid=1000 pid=23830 exe=/usr/bin/bash : path=/opt/fmrex/foreman-ssh-cmd-965edf46-51db-4bd4-bb3e-b220937ffb14/script-wrapper ftype=text/plain trust=0

So to fix:

echo "allow perm=execute exe=/usr/bin/bash trust=1 : dir=/opt/fmrex all trust=0" >> /etc/fapolicyd/rules.d/80-myapps.rules
fagenrules --load
systemctl start fapolicyd

Now REX works again. Hopefully this can help the next one coming across this issue

[Wes]

Thank you, @tedevil. It certainly did help me. Saved me a lot of time. Fapolicyd was truly the culprit in my case.

[tedevil]

Upgraded to Foreman/Katello 3.12/4.14 today and noticed this error when running remote execution on some hosts that had fapolicyd enabled:

unshare: failed to execute /opt/fmrex/foreman-ssh-cmd-86f24dcd-39ca-424c-b827-9b909a687af3/script-wrapper: Operation not permitted
Exit status: 126

debugging fapolicyd got me this block:

rule=17 dec=deny_audit perm=execute auid=1000 pid=55451 exe=/usr/bin/unshare : path=/opt/fmrex/foreman-ssh-cmd-2041e93f-ba89-4b19-a62f-d97c5164ef8a/script-wrapper ftype=text/plain trust=0

To enable remote execution working I added this rule:

allow perm=execute exe=/usr/bin/unshare trust=1 : dir=/opt/fmrex all trust=0

After this rule was added, remote execution started to work again.