Unable to change SSH keys for ssh_remote_execution

Unable to change SSH keys for ssh_remote_execution
I followed the instructions here
I tried to refresh the proxy as stated here
I even tried to reboot the host

The new public key is shown on https://localhost:8443/ssh/pubkey
But the old key is still in use when launching jobs

Expected outcome:
New SSH key used for ansible connection

Foreman and Proxy versions:
Foreman 2.3.1
Proxy 2.3.1
Ansible 6.0.1
Remote execution 4.2.1

Distribution and version:
CentOS 7

What exactly did you do? Do you have multiple proxies for remote execution?

I only have one smart proxy.

I generated a key:
ssh-keygen -t ed25519
Put it in the key directory with the same rights as the other key:
[root@ansible-foreman-centos ssh]# pwd
[root@ansible-foreman-centos ssh]# ls -l
total 12
-rw-------. 1 foreman-proxy foreman-proxy 452 8 janv. 09:31 id_ed25519_ocd-deployment
-rw-r--r--. 1 foreman-proxy foreman-proxy 130 8 janv. 09:36 id_ed25519_ocd-deployment.pub
Changed the proxy configuration file:
root@ansible-foreman-centos ssh]# cat /etc/foreman-proxy/settings.d/remote_execution_ssh.yml
:enabled: https
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_ed25519_ocd-deployment
:local_working_dir: /var/tmp
:remote_working_dir: /var/tmp
:kerberos_auth: false

# Whether to run remote execution jobs asynchronously
:async_ssh: false
Restarted the services and refreshed the proxy through the web interface (dynflowd from the doc doesn’t exist):
service httpd restart ; service foreman-proxy restart ; service smart_proxy_dynflow_core restart

I also tried to remove the old key files and make them a symlink to the new ones, and it works.
But I’d like to know why changing the parameter in the proxy conf file doesn’t work.

My guess would be because ansible has its own config file where the path to private key is defined and you most likely haven’t changed that one. Anything you set in /etc/foreman-proxy/settings.d/remote_execution_ssh.yml applies only to non-ansible execution with the ssh provider

Any clue about where that configuration could be ?
I can’t find it in /etc/ansible/ansible.cfg nor in /etc/foreman-proxy/ansible.cfg.

I even tried to grep id_rsa_foreman_proxy in all /etc and /usr and found nothing except foreman-installer configuration

I found it, it’s in the web interface, Settings -> Ansible -> Private key path
Many thanks @aruzicka for your help !

I looked at the code in the meantime and it looks like I was wrong. While it is overridable from the settings as you mentioned, if you don’t do that (and if I’m reading it right) it should inherit the configuration from /etc/foreman-proxy/settings.d/remote_execution_ssh.yml

From what I tested this is not the case…