I did a bit more debugging. The actual error does not show up in logs (a bug), I had to run smart_proxy_dynflow_core with logging to console to actually be able to see it.
E, [2020-06-15T03:20:30.722368 #7382] ERROR -- : OpenSSH keys only supported if ED25519 is available
net-ssh requires the following gems for ed25519 support:
* rbnacl (>= 3.2, < 5.0)
* rbnacl-libsodium, if your system doesn't have libsodium installed.
* bcrypt_pbkdf (>= 1.0, < 2.0)
See https://github.com/net-ssh/net-ssh/issues/478 for more information
Gem::MissingSpecError : "Could not find 'rbnacl' (< 5.0, >= 3.2.0) among 177 total gem(s)
Checked in 'GEM_PATH=/usr/share/foreman-proxy/.gem/ruby:/usr/share/gems:/usr/local/share/gems', execute `gem env` for more information"
(NotImplementedError)
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/ed25519_loader.rb:19:in `raiseUnlessLoaded'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/key_factory.rb:112:in `classify_key'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/key_factory.rb:52:in `load_data_private_key'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/key_factory.rb:43:in `load_private_key'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:142:in `sign'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/publickey.rb:62:in `authenticate_with'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/publickey.rb:20:in `block in authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:122:in `block in each_identity'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:119:in `each'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:119:in `each_identity'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:80:in `block in authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `each'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh.rb:241:in `start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:267:in `session'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:337:in `run_sync'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:426:in `ensure_remote_directory'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:403:in `upload_data'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:399:in `cp_script_to_remote'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:167:in `prepare_start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:153:in `start'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:32:in `start_runner'
Looking into why it only happens on EL8.