Remote Execution plugin + SSH Keys + Custom User?

Problem:
I would like to execute a command on a remote host using the ‘Run Command - Script Default’ template but I need to use my own ssh keys for that purpose and other username (not foreman-proxy).

So I copied my keys to: /usr/share/foreman-proxy/.ssh on my foreman server and replaced existing keys:

id_rsa_foreman_proxy
id_rsa_foreman_proxy.pub

And it doesn’t work. I can’t execute a command on that host because It fails immediately with:

Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried password
   2:
Exit status: EXCEPTION
   3:
StandardError: Job execution failed

Is it possible? If yes, what I am doing wrong?

Expected outcome:
Foreman should execute commands on a remote hosts using ssh keys and provided username

Foreman and Proxy versions:
Foreman 3.8, latest

Distribution and version:
Ubuntu 20.04

Hi,
did you correctly set permissions when you placed the keys into ~foreman-proxy/.ssh? Is your key passphrase protected?

I don’t recommend swapping the ssh keys out from under the foreman-proxy users feet. I don’t think those are used for just remote execution.

You should really create a completely separate remote execution user (e.g. rex-user) with the keys you want, and then configure remote execution to use that user by setting the remote_execution_ssh_user parameter to (type String, value rex-user) for any relevant hosts. Of course the rex user you create needs to own the ssh keys you want to use, etc.

Just to rule out common issues, which are not remote execution related:

Using the key you placed inside your foreman, can you open a ssh-connection to your target host on the terminal? Did you add the pub-key to the authorized_keys on the target host?

On the foreman the key has to be readable by foreman-proxy (only → private key shouldn’t be readable for other users)

On the foreman the ssh-connection is always opened by the foreman-proxy-user but you can specify which user should be used to login at the targetsystem (default: root) but you can specify the username.

might also need to look at the foreman-installer settings depending on how “custom” you want to make it:
–foreman-proxy-plugin-remote-execution-script-local-working-dir
–foreman-proxy-plugin-remote-execution-script-ssh-identity-dir
–foreman-proxy-plugin-remote-execution-script-ssh-identity-file
Any wrong moves here though and you will start to break things so be careful and snapshot before :slight_smile:

2 Likes

Hi,
did you correctly set permissions when you placed the keys into ~foreman-proxy/.ssh? Is your key passphrase protected?

Yes, permissions are correct.
No, it is not password protected.

You should really create a completely separate remote execution user (e.g. rex-user) with the keys you want, and then configure remote execution to use that user by setting the remote_execution_ssh_user parameter to (type String, value rex-user) for any relevant hosts. Of course the rex user you create needs to own the ssh keys you want to use, etc.

Okay, I reverted my changes, so original foreman-proxy generated keys and restarted foreman. I can change the username in remote_execution_ssh_user field, but how foreman will find ssh key for that user?

Using the key you placed inside your foreman, can you open a ssh-connection to your target host on the terminal? Did you add the pub-key to the authorized_keys on the target host?

Yes, I was able to ssh by providing the ssh key as an argument e.g. ssh hostname -i key

might also need to look at the foreman-installer settings depending on how “custom” you want to make it:
–foreman-proxy-plugin-remote-execution-script-local-working-dir
–foreman-proxy-plugin-remote-execution-script-ssh-identity-dir
–foreman-proxy-plugin-remote-execution-script-ssh-identity-file

I decided to leave original foreman-proxy user name as it is. I want to use foreman-proxy one.

I’m not following. The proxy has a key. The key stays the same no matter what user on the remote end you want to use

Hello

I wanted to use another key, not foreman-proxy key as well as other user.
Fortunately, I managed to do this by changing the ssh config for foreman-proxy user. This is how you can do it:

  1. Create config file in:
    /usr/share/foreman-proxy/.ssh/config

  2. Add following configuration:

Host *
    IdentityFile /path/to/your/non-foreman-proxy.public-key
  1. Restart sshd
    systemctl restart sshd

  2. Add an extra parameter for a host or a global parameter:
    remote_execution_ssh_user = YOUR_CUSTOM_NON-FOREMAN-PROXY-USER

That’s it. It’s going to work.

1 Like

Yes, that should work as of now, but it is somewhat non-standard way of doing things.

What would be the standard way of doing this?

We’re bound by compliance requirements so the default RSA key is less than desirable for us. I tried @cps86 's fix and it resolved our issue when trying to use an ECDSA key. If there’s a more standard was I’d prefer to use that. Does the installer typically copy the ECDSA key into place if it’s specified upon initial install?

What tedevil suggested a couple of posts above would be cleaner than configuring ssh itself. The installer has options to deal with situations like this.

I dug back into my configuration, as I had indeed utilized @tedevil 's method, but with seemingly no effect from it. It appears that despite providing the --foreman-proxy-plugin-remote-execution-script-ssh-identity-file parameter to the installer, with id_ecdsa_foreman_proxy as an argument, running foreman-installer --full-help yield’s a current setting of still the default id_rsa_foreman_proxy. I’m wondering if there is some kind of issue that might be preventing the setting from actually being applied.

That’s odd, there shouldn’t be anything that could make the installer decide to ignore the value you pass in. The installer should take the values for --foreman-proxy-plugin-remote-execution-script-ssh-identity-dir and --foreman-proxy-plugin-remote-execution-script-ssh-identity-dir, join them with a forward slash and write that out under a specific key into /etc/foreman-proxy/settings.d/remote_execution_ssh.yml.

I tried adding in --foreman-proxy-plugin-remote-execution-script-ssh-identity-dir, just to be sure it was picking up my options, but it still resulted in no change visible to the current setting in the full help output. I also checked the katello-answers.yaml file, as well as the remote_execution_ssh.yml files, with no luck seeing any change in output. Is it perhaps an option that can only be specified on the very first installer run?

Looks like the only way to do this is by modifying the /etc/foreman-proxy/settings.d/remote_execution_ssh.yml config file, and changing the default settings there:

:enabled: https
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
:local_working_dir: /var/tmp
:remote_working_dir: /var/tmp
:kerberos_auth: false

Not from what I’ve seen. It seems to work on my box no matter when I pass it in.

Well, that’s generated by the installer so chances are it will stomp over it the next time it runs.