SSH Remote Execution (REX) broken after `dnf update`

Support
1

Problem:
After running

  • dnf update
  • reboot
  • foreman-installer

All SSH Remote Executions in Foreman are broken, and don’t work anymore.

Error:

Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried publickey

Expected outcome:
dnf update should not break anything in Foreman, nor underlying ssh, and if it does, foreman-installer should repair it.

Foreman and Proxy versions:
3.5.1

Foreman and Proxy plugin versions:
katello 4.7.2
foreman-tasks 7.1.1
foreman_remote_execution 8.2.0

Distribution and version:
Rocky Linux 8.7

Other relevant data:
I should note that I had previously manually created and added a new SSH key pair in EdDSA format (Ed25519).
It worked flawlessly before the dnf update.

Start any REX job, e.g.:

image
image1002×234 29.3 KB

The REX job, and the resulting error:
image

2

foreman-installer has reversed a manual change that I had done to switch from RSA to EdDSA SSH identity files.
This is the affected file:

/etc/foreman-proxy/settings.d/remote_execution_ssh.yml

Presumably, this problem will not affect other users widely, but only me, or very few users.

It seems that manual changes to Foreman config files shouldn’t be done directly on config files, as they will be overwritten by the next run of foreman-installer.

Is there a better, correct way of altering config files, where changes will remain preserved by foreman-installer?

image
image695×215 30.3 KB

3

Usually using foreman-installer parameters. In this case: --foreman-proxy-plugin-remote-execution-script-ssh-identity-file ? See foreman-installer --full-help.

1 Like
4

Many thanks @JendaVodka , I agree your suggested solution is the best way to make this change.

However, the documentation isn’t quite clear how to specify the ssh key file. If the full path is used, the installed errors out. The configuration script really is a bit obscure, but it does work.

# foreman-installer --foreman-proxy-plugin-remote-execution-script-ssh-identity-file id_ed25519_foreman_proxy
2023-03-06 18:11:12 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-03-06 18:11:14 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-03-06 18:11:14 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-03-06 18:11:19 [NOTICE] [configure] Starting system configuration.
2023-03-06 18:11:26 [NOTICE] [configure] 250 configuration steps out of 1457 steps complete.
2023-03-06 18:11:28 [NOTICE] [configure] 500 configuration steps out of 1459 steps complete.
2023-03-06 18:11:30 [NOTICE] [configure] 750 configuration steps out of 1464 steps complete.
2023-03-06 18:11:31 [NOTICE] [configure] 1000 configuration steps out of 1468 steps complete.
2023-03-06 18:11:48 [NOTICE] [configure] 1250 configuration steps out of 1468 steps complete.
2023-03-06 18:11:59 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
Processing Repository 1/110: centos7-x86_64-extras (1)
-- x --- snip --- x --
Processing Repository 109/110: Rocky Linux 9 - Extras (x86) (537)
Processing Repository 110/110: epel9 (538)
=============================================
Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
  Success!
  * Foreman is running at   https://foreman.my.org
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  * Foreman Proxy is running at   https://foreman.my.org:9090

  The full log is at /var/log/foreman-installer/katello.log