Remote execution give me "Operation not permitted" from script-wrapper

tedevil · July 30, 2022, 9:28pm

Running on Foreman 3.3/Katello 4.5.
Have started to run some tests on a hardened AlmaLinux 9 and for some reason I get this error from a remote execution run:

 1: sh: line 1: /opt/fmrex/foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2/script-wrapper: Operation not permitted
 2: Exit status: 126

The command sent was just “date” so I expected no problems.
If I log in manually from the proxy, I can log in with ssh using the ~foreman-proxy/.ssh/id_rsa_foreman_proxy key, use sudo to elevate myself to root and issue the command without problems so I am a little confused what is not allowed.

I fetched the script-wrapper file and it contains:

echo $$ > /opt/fmrex/foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2/pid; exec "$@";

Not totally sure what these commands are supposed to do but as root I have no problems executing them.
Logging however show the pid file was never created during remote execution so this command is my suspect. Anyone knows the details on how this is working. Was left with this evidence:

[root@sees-almatest foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2]# ls -la
total 20
drwxr-x---. 2 root  root  112 Jul 30 21:26 .
drwxr-x---. 3 fmrex fmrex  66 Jul 30 20:58 ..
-rw-r-----. 1 root  root   51 Jul 30 20:58 effective-user-test
-rw-r-----. 1 root  root    4 Jul 30 20:58 exit_code
-rw-r-----. 1 root  root    0 Jul 30 20:58 output
-rw-r-----. 1 root  root    5 Jul 30 20:58 script
-rw-r-----. 1 root  root   90 Jul 30 20:58 script-wrapper
-rw-r-----. 1 root  root   20 Jul 30 20:58 test
[root@sees-almatest foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2]# grep . *
effective-user-test:#!/bin/sh
effective-user-test:exec sudo -p 'rex login: ' -u root  true
exit_code:126
script:date
script-wrapper:echo $$ > /opt/fmrex/foreman-ssh-cmd-e77aa532-a5d4-4b18-9004-a97f540427f2/pid; exec "$@";
test:#!/bin/sh
test:exec true

rplevka · July 30, 2022, 10:13pm

Just a hint, that the exit code 126 indicates that the file it is trying to execute is missing the executable flag - it’s also visible in the file listing you pasted.

tedevil · July 30, 2022, 10:25pm

Did the same file pulling from a non hardeded AlmaLinux 8 and it shows they have indeed other permissions:

[fmrex@sees-fmtestvm2 foreman-ssh-cmd-6e497bc4-bab1-4188-8938-cfa5df039d1f]$ ls -la
total 28
drwxr-xr-x. 2 root root 123 Jul 30 21:24 .
drwxr-xr-x. 9 root root 265 Jul 30 21:24 ..
-r-xr-xr-x. 1 root root  51 Jul 30 21:24 effective-user-test
-rw-r--r--. 1 root root   2 Jul 30 21:24 exit_code
-rw-r--r--. 1 root root  29 Jul 30 21:24 output
-rw-r--r--. 1 root root   7 Jul 30 21:24 pid
-r-xr-xr-x. 1 root root   5 Jul 30 21:24 script
-r-xr-xr-x. 1 root root  90 Jul 30 21:24 script-wrapper
-r-xr-xr-x. 1 root root  20 Jul 30 21:24 test
[fmrex@sees-fmtestvm2 foreman-ssh-cmd-6e497bc4-bab1-4188-8938-cfa5df039d1f]$ grep . *
effective-user-test:#!/bin/sh
effective-user-test:exec sudo -p 'rex login: ' -u root  true
exit_code:0
output:Sat Jul 30 21:24:15 UTC 2022
pid:244690
script:date
script-wrapper:echo $$ > /opt/fmrex/foreman-ssh-cmd-6e497bc4-bab1-4188-8938-cfa5df039d1f/pid; exec "$@";
test:#!/bin/sh
test:exec true

How is remote executing setting the permissions on these files?

rplevka · July 31, 2022, 9:21am

Hi.
I believe it is uploading the files with permissions 555 here:

github.com

theforeman/smart_proxy_remote_execution_ssh/blob/master/lib/smart_proxy_remote_execution_ssh/runners/script_runner.rb#L183

    
      
                                    user_method: @user_method,
                                    close_stdin: false)
            end
          end
          
          
def prepare_start
            @remote_script = cp_script_to_remote
            @output_path = File.join(File.dirname(@remote_script), 'output')
            @exit_code_path = File.join(File.dirname(@remote_script), 'exit_code')
            @pid_path = File.join(File.dirname(@remote_script), 'pid')
            @remote_script_wrapper = upload_data("echo $$ > #{@pid_path}; exec \"$@\";", File.join(File.dirname(@remote_script), 'script-wrapper'), 555)
          end
          
          
# the script that initiates the execution
          def initialization_script
            su_method = @user_method.instance_of?(SuUserMethod)
            # pipe the output to tee while capturing the exit code in a file
            <<~SCRIPT
              sh <<EOF | /usr/bin/tee #{@output_path}
              #{@remote_script_wrapper} #{@user_method.cli_command_prefix}#{su_method ? "'#{@remote_script} < /dev/null '" : "#{@remote_script} < /dev/null"}
              echo \\$?>#{@exit_code_path}

Anyway, maybe the hardening mounts the destination with noexec flag?
If that is true, you might need to reconfigure REX to use different working path, just like described here:

Hope that helps!

tedevil · July 31, 2022, 2:14pm

The partition has not the noexec flag (first thing I checked ).

Anyway, was digging around in the “OpenSCAP Evaluation Report” and came across an activation of fapolicyd “(File Access Policy Daemon) implements application whitelisting to decide file access rights”. Seen it was enabled. Stopped it and ran “fapolicyd --debug-deny”, ran the rex command again and got:

rule=13 dec=deny_audit perm=execute auid=1000 pid=23830 exe=/usr/bin/bash : path=/opt/fmrex/foreman-ssh-cmd-965edf46-51db-4bd4-bb3e-b220937ffb14/script-wrapper ftype=text/plain trust=0

So to fix:

echo "allow perm=execute exe=/usr/bin/bash trust=1 : dir=/opt/fmrex all trust=0" >> /etc/fapolicyd/rules.d/80-myapps.rules
fagenrules --load
systemctl start fapolicyd

Now REX works again. Hopefully this can help the next one coming across this issue

Wes · April 24, 2023, 6:43pm

Thank you, @tedevil. It certainly did help me. Saved me a lot of time. Fapolicyd was truly the culprit in my case.