Remote Executtion has stopped working

Support
1

Problem:
Remote execution has stopped working. We don’t use it often, but in some circumstances it is needed to push out a package or some other task. Logs show the remote execution user logging in fine, and I can see in the logs: OMMAND=/var/remoteexec/foreman-ssh-cmd-5d483e93-775c-4711-af4f-b940d8c164a9/script. However, when I tried running again, and watching for the creation of a foreman command. I never see it appear.

Here are the permission for our execution directory, /var/remoteexec:

drwxr-xr-x. 2 foreman-proxy root 6 Oct 10 13:41 remoteexec

I have used remote execution many times in the past, and has always worked. I haven’t used it recently, so I’m not sure if this is something introduce in Foreman 3.11 or earlier because I don’t know exactly when this stopped working.

I also noticed that the logs below show that the login and logout are fairly quick while the TASK on Foreman is only showing 25% when the logout occurs.

Expected outcome:

Remote execution commands should work. Simple jobs like “ls” or “df” are no longer working.

Foreman and Proxy versions:

Foreman 3.11.2
Katello 4.13.1

Foreman and Proxy plugin versions:

foreman-tasks 9.1.1
foreman_discovery 24.0.1
foreman_remote_execution 13.1.0
katello 4.13.1

Distribution and version:

Red Hat Enterprise Linux release 8.10 (Ootpa)

Other relevant data:

I couldn’t find any specific logs on the foreman server or proxy that tries to run the job.

Job Status:

1:Exit status: 0
2:RuntimeError: Unexpected event #<Actions::ProxyAction::ProxyActionStopped: Actions::ProxyAction::ProxyActionStopped>

/var/log/secure on server where execuion is running
Oct 10 11:34:53 server1 sshd[2592011]: Accepted publickey for foreman-proxy from 172.21.240.2 port 55834 ssh2: RSA SHA256:ouBNh+B5r8q7Zt3ca/xqaklvVJCJ7pA6/DYvrESbB3Y
Oct 10 11:34:53 server1 systemd[2592016]: pam_unix(systemd-user:session): session opened for user foreman-proxy by (uid=0)
Oct 10 11:34:53 server1 sshd[2592011]: pam_unix(sshd:session): session opened for user foreman-proxy by (uid=0)
Oct 10 11:34:53 server1 sudo[2592031]: foreman-proxy : TTY=pts/2 ; PWD=/home/foreman-proxy ; USER=root ; COMMAND=/var/remoteexec/foreman-ssh-cmd-5d483e93-775c-4711-af4f-b940d8c164a9/script
Oct 10 11:34:53 server1 sudo[2592031]: pam_unix(sudo:session): session opened for user root by foreman-proxy(uid=1000)
Oct 10 11:34:54 server1 sudo[2592031]: pam_unix(sudo:session): session closed for user root
Oct 10 11:34:54 aka01adi11a sudo[2592051]: foreman-proxy : TTY=pts/2 ; PWD=/home/foreman-proxy ; USER=root ; COMMAND=/var/remoteexec/foreman-ssh-cmd-5d483e93-775c-4711-af4f-b940d8c164a9/script
Oct 10 11:34:54 server1 sudo[2592051]: pam_unix(sudo:session): session opened for user root by foreman-proxy(uid=1000)
Oct 10 11:35:03 server1 sudo[2592051]: pam_unix(sudo:session): session closed for user root
Oct 10 11:35:03 server1 sshd[2592025]: Received disconnect from 172.21.240.2 port 55834:11: disconnected by user
Oct 10 11:35:03 server1 sshd[2592025]: Disconnected from user foreman-proxy 172.21.240.2 port 55834
Oct 10 11:35:03 server1 sshd[2592011]: pam_unix(sshd:session): session closed for user foreman-proxy

2

See https://community.theforeman.org/t/foreman-3-11-1-remoteexecution-runtimeerror . The error you’re getting is a side effect of the proxy->foreman callback not going through.

3

Looking at the URL, I would suspect is the certs, since we have recently had to update, and go with a different CA. I have to do this yearly for certs (usually just to renew, not change CAs). However, the flags I’m using for Foreman installer don’t seem to be updating all the certs. Specifiically, /etc/pki/katello/certs/katello-apache.crt and /etc/pki/katello/private/katello-apache.key.

Here are the flags I’m using on the foreman master and the proxies:

Master:
foreman-installer --scenario katello
–certs-server-cert “/etc/pki/tls/certs/master.crt”
–certs-server-key “/etc/pki/tls/private/master.key”
–certs-server-ca-cert “/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem”
–certs-update-server

Proxies:
foreman-installer --certs-tar-file “/root/proxy1-certs.tar” --certs-update-all --certs-regenerate true --certs-deploy true

Can you tell me what I’m missing? This has always worked correctly in the past.

4

Generally, I would advise to follow the docs.

https://docs.theforeman.org/3.11/Administering_Project/index-katello.html#renewing-the-custom-ssl-certificate_admin

The ca cert is supposed to be the chain of certificates which issued the server certificate, i.e. the issuing ca of the server ca, intermediates and the root. Don’t set it to trusted bundle. It’s missing the intermediates, if any, and secondly makes foreman basically to accept any certificate by any ca, including public ones like lets encrypt.

katello-certs-check will print the foreman-installer options to be used including those shown in the example in the docs.

However, there is a bug in 3.11 which has been fixed but not ported to 3.11, yet:

Thus, you probably have to manually fix this until it is included in 3.11 or you have upgraded to 3.12.