Unable to change SSH keys for ssh_remote_execution

Support
#1

Problem:
Unable to change SSH keys for ssh_remote_execution
I followed the instructions here
I tried to refresh the proxy as stated here
I even tried to reboot the host

The new public key is shown on https://localhost:8443/ssh/pubkey
But the old key is still in use when launching jobs

Expected outcome:
New SSH key used for ansible connection

Foreman and Proxy versions:
Foreman 2.3.1
Proxy 2.3.1
Ansible 6.0.1
Remote execution 4.2.1

Distribution and version:
CentOS 7

#2

What exactly did you do? Do you have multiple proxies for remote execution?

#3

I only have one smart proxy.

I generated a key:
ssh-keygen -t ed25519
Put it in the key directory with the same rights as the other key:
[root@ansible-foreman-centos ssh]# pwd
/var/lib/foreman-proxy/ssh
[root@ansible-foreman-centos ssh]# ls -l
total 12
-rw-------. 1 foreman-proxy foreman-proxy 452 8 janv. 09:31 id_ed25519_ocd-deployment
-rw-r--r--. 1 foreman-proxy foreman-proxy 130 8 janv. 09:36 id_ed25519_ocd-deployment.pub
Changed the proxy configuration file:
root@ansible-foreman-centos ssh]# cat /etc/foreman-proxy/settings.d/remote_execution_ssh.yml
---
:enabled: https
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_ed25519_ocd-deployment
:local_working_dir: /var/tmp
:remote_working_dir: /var/tmp
:kerberos_auth: false

# Whether to run remote execution jobs asynchronously
:async_ssh: false
Restarted the services and refreshed the proxy through the web interface (dynflowd from the doc doesn’t exist):
service httpd restart ; service foreman-proxy restart ; service smart_proxy_dynflow_core restart

I also tried to remove the old key files and make them a symlink to the new ones, and it works.
But I’d like to know why changing the parameter in the proxy conf file doesn’t work.

#4

My guess would be because ansible has its own config file where the path to private key is defined and you most likely haven’t changed that one. Anything you set in /etc/foreman-proxy/settings.d/remote_execution_ssh.yml applies only to non-ansible execution with the ssh provider

#5

Any clue about where that configuration could be ?
I can’t find it in /etc/ansible/ansible.cfg nor in /etc/foreman-proxy/ansible.cfg.

I even tried to grep id_rsa_foreman_proxy in all /etc and /usr and found nothing except foreman-installer configuration

#6

I found it, it’s in the web interface, Settings -> Ansible -> Private key path
Many thanks @aruzicka for your help !

#7

I looked at the code in the meantime and it looks like I was wrong. While it is overridable from the settings as you mentioned, if you don’t do that (and if I’m reading it right) it should inherit the configuration from /etc/foreman-proxy/settings.d/remote_execution_ssh.yml

#8

From what I tested this is not the case…