SSH Remote Execution (REX) broken after `dnf update`

After running

  • dnf update
  • reboot
  • foreman-installer

All SSH Remote Executions in Foreman are broken, and don’t work anymore.


Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried publickey

Expected outcome:
dnf update should not break anything in Foreman, nor underlying ssh, and if it does, foreman-installer should repair it.

Foreman and Proxy versions:

Foreman and Proxy plugin versions:
katello 4.7.2
foreman-tasks 7.1.1
foreman_remote_execution 8.2.0

Distribution and version:
Rocky Linux 8.7

Other relevant data:
I should note that I had previously manually created and added a new SSH key pair in EdDSA format (Ed25519).
It worked flawlessly before the dnf update.

Start any REX job, e.g.:

The REX job, and the resulting error:

foreman-installer has reversed a manual change that I had done to switch from RSA to EdDSA SSH identity files.
This is the affected file:


Presumably, this problem will not affect other users widely, but only me, or very few users.

It seems that manual changes to Foreman config files shouldn’t be done directly on config files, as they will be overwritten by the next run of foreman-installer.

Is there a better, correct way of altering config files, where changes will remain preserved by foreman-installer?

Usually using foreman-installer parameters. In this case: --foreman-proxy-plugin-remote-execution-script-ssh-identity-file ? See foreman-installer --full-help.

1 Like

Many thanks @JendaVodka , I agree your suggested solution is the best way to make this change.

However, the documentation isn’t quite clear how to specify the ssh key file. If the full path is used, the installed errors out. The configuration script really is a bit obscure, but it does work.

# foreman-installer --foreman-proxy-plugin-remote-execution-script-ssh-identity-file id_ed25519_foreman_proxy
2023-03-06 18:11:12 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-03-06 18:11:14 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-03-06 18:11:14 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-03-06 18:11:19 [NOTICE] [configure] Starting system configuration.
2023-03-06 18:11:26 [NOTICE] [configure] 250 configuration steps out of 1457 steps complete.
2023-03-06 18:11:28 [NOTICE] [configure] 500 configuration steps out of 1459 steps complete.
2023-03-06 18:11:30 [NOTICE] [configure] 750 configuration steps out of 1464 steps complete.
2023-03-06 18:11:31 [NOTICE] [configure] 1000 configuration steps out of 1468 steps complete.
2023-03-06 18:11:48 [NOTICE] [configure] 1250 configuration steps out of 1468 steps complete.
2023-03-06 18:11:59 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
Processing Repository 1/110: centos7-x86_64-extras (1)
-- x --- snip --- x --
Processing Repository 109/110: Rocky Linux 9 - Extras (x86) (537)
Processing Repository 110/110: epel9 (538)
Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
  * Foreman is running at
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  * Foreman Proxy is running at

  The full log is at /var/log/foreman-installer/katello.log