Ansible fails to run role from GUI

bughatti · September 3, 2019, 8:57pm

I just recently installed foreman and have also added ansible to it. I am trying to get ansible to run some basic stuff but for some reason does not want to run. I always get

msg: ‘Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).’

doing research, I have made sure to copy the ssh keys from the foreman server to the host I want to run ansible against. I have tried a different user other than root. I followed instructions to check on the permissions of .ansible from this post Running playbook role in Foreman 1.15.6 / ansible pluging 1.4.5 / ansible 2.4.1 return an error which mine was a bit different as it was under foreman-proxy folder, not foreman folder. I tried to change the permissions but it is not taking the command. I also turned on logging for ansible. After logging is on, I can run ansible -m ping pdm-wiki -vvv from the foreman server and it works fine against the other server and it will log to the ansible log. When i press run in the foreman gui for a change of the ntp server within geerlingguy.ntp I get the error above and it does not log anything to ansible.log at all.

I am using foreman 1.22 with ansible 2.1.2

scottc · September 4, 2019, 3:31pm

Hi @bughatti,
I think the Ansible plugin needs the ssh keys to be in ~foreman-proxy/.ssh/, not ~foreman/.ssh/ . I think that’s required for other remote execution stuff to work too. Also check that they’re readable by the foreman-proxy user.

(I’m not a dev, but that’s how I’ve got it set up.)

Marek_Hulan · September 4, 2019, 4:00pm

This is correc, thanks for sharing. The connection is established by the proxy that runs under foreman-proxy user.

bughatti · September 4, 2019, 4:27pm

So looking in my system I have a

ls -al ~/.ssh/
id_rsa id_rsa.pub known_hosts

and

ls -al ~foreman/.ssh/
total 4
drwx------. 2 foreman foreman 6 Jun 4 08:04 .
drwxr-xr-x. 15 root root 4096 Sep 3 14:16 …

and also

ls -al ~foreman-proxy/.ssh/
total 12
drwxr-x—. 2 foreman-proxy foreman-proxy 85 Sep 3 09:50 .
drwxr-xr-x. 7 foreman-proxy foreman-proxy 85 Sep 3 08:49 …
-rw-------. 1 foreman-proxy foreman-proxy 1679 Sep 3 08:49 id_rsa_foreman_proxy
-rw-r–r--. 1 foreman-proxy foreman-proxy 420 Sep 3 08:49 id_rsa_foreman_proxy.pub
-rw-r–r--. 1 foreman-proxy foreman-proxy 1667 Sep 3 10:51 known_hosts

what resides in ~/.ssh/ is different than what is in ~foreman-proxy/.ssh

scottc · September 4, 2019, 4:32pm

Looking at my system id_rsa_foreman_proxy is linked to id_rsa, and ditto for id_rsa.pub, so

cd ~foreman-proxy/.ssh
ln -s id_rsa_foreman_proxy id_rsa
ln -s id_rsa_foreman_proxy.pub id_rsa.pub

…should get you working, I think (assuming that public key is on the hosts to be managed).

bughatti · September 4, 2019, 4:50pm

Ok so still getting

PLAY [all] *********************************************************************
3:

4:
TASK [Gathering Facts] *********************************************************
5:
fatal: [pdm-wiki.xxxx.local]: UNREACHABLE! => changed=false
6:
msg: ‘Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).’
7:
unreachable: true
8:

9:
PLAY RECAP *********************************************************************
10:
pdm-wiki.xxxx.local : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
11:

12:
Exit status: 4

Maybe I am missing something completely different here. So I have foreman setup and working, I have been able to push NTP updates via puppet. With ansible installed, I am able to go to my host, select edit > Ansible Roles and add geerlingguy.ntp to the host. The only variable I am using from that module is to set ntp servers.

I go back to host, select run ansible roles and it fails with the above error. I am using root as the Ansible Roles - Ansible Default effective user

If I go back to cli on my foreman host logged in as root, I can effectively ssh root@othermanagedhost and connect fine.

scottc · September 4, 2019, 5:00pm

… but it’s the foreman-proxy user on your Foreman server and the id_rsa_foreman_proxy.pub user (=root in your case) on othermanagedhost that are in play when Foreman tries to run Ansible.

As root on your foreman host, does ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@othermanagedhost work?

If you su to foreman-proxy, can you ssh to root@othermanagedhost?

(foreman-proxy might not have a valid shell in /etc/passwd, so you might have to usermod -s it (or whatever your local systems require) for a few minutes during testing.

scottc · September 4, 2019, 5:02pm

Now I look a bit further, I see I have a ~foreman-proxy/.ssh/ssh_config file as well.

BatchMode yes
CheckHostIP no
StrictHostKeyChecking no

Maybe it was getting stuck on the first connection host fingerprint check. (I should check that on my own system.)

bughatti · September 4, 2019, 5:06pm

so doing the below works, but it asks for the password.

ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@pdm-wiki

Also when i do su foreman-proxy it will not switch to that account
[root@pdm-foreman ~]# su foreman-proxy
[root@pdm-foreman ~]#

I am using centos 7

scottc · September 4, 2019, 5:11pm

That would suggest to me that you don’t have the contents of ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub in pdm-wiki:~root/.ssh/authorized_keys.

Or you have Some Other SSH Issue.

I’m reluctant to get into debugging an SSH-specific issue with you, because of the obvious security implications.

But I think you need to make that line work, without a password prompt, to make the Ansible work.

scottc · September 4, 2019, 5:14pm

The shell for foreman-proxy is probably (correctly) set to /bin/false in /etc/passwd. That’s intentional for security reasons, but it means you can’t just su over to the account.

bughatti · September 4, 2019, 5:21pm

so I am new to some of this stuff, basics I can usually accomplish but getting in depth I have to dig around. All i did to copy the keys was from the foreman terminal I did ssh-copy-id root@pdm-wiki

something i stated earlier before doing the ln commands, the contents of ~/.ssh and ~foreman-proxy/.ssh had been different, maybe doing the ln changed something, but even after the ln commands, I still went back and ran ssh-copy-id and it still does not work. I can redo keys if that is necessary, will just need to lookup what commands to run

scottc · September 4, 2019, 5:29pm

I think you probably just have to repeat your ssh-copy-id command, but add -i ~foreman-proxy/.ssh/id_rsa, so it sends the foreman-proxy key, not the key for the user that you’re running as on the foreman server.

But it’s important that you understand what you’re doing, because messing with auth can get you into trouble. “Any advice given is for novelty value only. If in doubt, consult a doctor.”

(sorry - I have to go now, but I’ll check back later tonight or tomorrow)

bughatti · September 4, 2019, 5:34pm

That was it, thanks a ton for the help. In all the docs i stumbled across for setting up foreman and ansible together, none really went over this.

scottc · September 5, 2019, 9:46am

For future reference,

The ln -s commands I suggested in reply 4 don’t seem to be required. I may have had them on my system for debugging purposes. Ansible plugin v3.0.2 runs without them.
The ssh_config file from reply 7 is convenient if your users frequently rebuild hosts in your environment (e.g. dev, test), resulting in changed host keys, but you don’t want to have to clean out ~foreman-proxy/.ssh/known_hosts manually. Otherwise, you probably shouldn’t have it - it removes security checks from ssh.