Can't run ansible roles

Support
#1

Hello everybody,

My problem is i can’t run ansible roles via Foreman WebUI, you can show error in screenshot

ansible_role
ansible_role.PNG1718×628 34.8 KB

Ansible callback works.

Ansible version : 2.4.2.0
Foreman version : 1.18.0.0 (i have same problem with 1.16. version)
Katello version : 3.7 (i have same problem with 3.5 version)

The problem surely come from me but i have not yet found the solution.

This command works :

1st command:

bash-4.2$ whoami
foreman
bash-4.2$ sudo ansible-playbook test.yml

PLAY [foreman.mydomain.lab] ******************************************************************************

TASK [Gathering Facts] *****************************************************************************************
ok: [foreman.mydomain.lab]

TASK [Create file] *********************************************************************************
changed: [foreman.mydomain.lab]

PLAY RECAP **********************************************************************************
foreman.mydomain.lab         : ok=2    changed=1    unreachable=0    failed=0

2nd command:

bash-4.2$ whoami
foreman-proxy
bash-4.2$ sudo ansible-playbook test.yml

PLAY [foreman.mydomain.lab] ***************************************************************************************

TASK [Gathering Facts] **************************************************************************************

ok: [foreman.mydomain.lab]

TASK [Create file] *******************************************************************************************
changed: [foreman.mydomain.lab]

PLAY RECAP ******************************************************************************************

foreman.mydomain.lab         : ok=2    changed=1    unreachable=0    failed=0

Logs :

2018-02-15T16:44:56 6e4072a9 [app] [I] Started GET "/ansible/hosts/6/play_roles" for 172.23.10.4 at 2018-02-15 16:44:56 +0100
2018-02-15T16:44:56 6e4072a9 [app] [I] Processing by HostsController#play_roles as HTML
2018-02-15T16:44:56 6e4072a9 [app] [I]   Parameters: {"id"=>"6"}
2018-02-15T16:44:56 6e4072a9 [app] [I] Current user: admin (administrator)
2018-02-15T16:44:56 6e4072a9 [app] [D] Setting current user thread-local variable to admin
2018-02-15T16:44:56 6e4072a9 [app] [D] Setting current location thread-local variable to none
2018-02-15T16:44:56 6e4072a9 [app] [D] Setting current organization thread-local variable to none
2018-02-15T16:44:57 6e4072a9 [dynflow] [D] ExecutionPlan 7fe9991d-c609-4e64-882e-97df12c58152      pending >>  planning
2018-02-15T16:44:57 6e4072a9 [dynflow] [D]          Step 7fe9991d-c609-4e64-882e-97df12c58152: 1   pending >>   running in phase     Plan Actions::RemoteExecution::RunHostsJob
2018-02-15T16:44:57 6e4072a9 [app] [I] Current user: admin (administrator)
2018-02-15T16:44:57 6e4072a9 [app] [D] Setting current user thread-local variable to admin
2018-02-15T16:44:57 6e4072a9 [app] [I] Current user: admin (administrator)
2018-02-15T16:44:57 6e4072a9 [app] [D] Setting current user thread-local variable to admin
2018-02-15T16:44:57 6e4072a9 [dynflow] [D]          Step 7fe9991d-c609-4e64-882e-97df12c58152: 1   running >>   success in phase     Plan Actions::RemoteExecution::RunHostsJob
2018-02-15T16:44:57 6e4072a9 [dynflow] [D] ExecutionPlan 7fe9991d-c609-4e64-882e-97df12c58152     planning >>   planned
2018-02-15T16:44:57 6e4072a9 [app] [I] Redirected to htps://foreman.mydomain.lab/job_invocations/37
2018-02-15T16:44:57 6e4072a9 [app] [I] Completed 302 Found in 1067ms (ActiveRecord: 155.2ms)
2018-02-15T16:44:57 1d8d4696 [app] [I] Started GET "/job_invocations/37" for 172.23.10.4 at 2018-02-15 16:44:57 +0100
2018-02-15T16:44:57 1d8d4696 [app] [I] Processing by JobInvocationsController#show as HTML
2018-02-15T16:44:57 1d8d4696 [app] [I]   Parameters: {"id"=>"37"}
2018-02-15T16:44:57 1d8d4696 [app] [I] Current user: admin (administrator)
2018-02-15T16:44:57 1d8d4696 [app] [D] Setting current user thread-local variable to admin
2018-02-15T16:44:57 1d8d4696 [app] [D] Setting current location thread-local variable to none
2018-02-15T16:44:57 1d8d4696 [app] [D] Setting current organization thread-local variable to none
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/show.html.erb within layouts/application
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_tab_overview.html.erb (91.1ms)
2018-02-15T16:44:57 1d8d4696 [app] [D] Unpermitted parameter: :id
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_host_name_td.html.erb (1.2ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_host_status_td.html.erb (1.0ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_host_actions_td.html.erb (2.5ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered common/_pagination.html.erb (2.4ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_tab_hosts.html.erb (17.7ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/show.html.erb within layouts/application (191.8ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered layouts/_application_content.html.erb (1.1ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendering layouts/base.html.erb
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_organization_dropdown.html.erb (30.2ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_location_dropdown.html.erb (16.5ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_org_switcher.html.erb (49.8ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_user_dropdown.html.erb (2.5ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_topbar.html.erb (56.2ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (6.3ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (11.7ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (1.6ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (5.7ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (4.6ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (3.4ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (4.8ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_taxonomies.html.erb (3.6ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_taxonomies.html.erb (2.6ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_vertical_menu.html.erb (0.9ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered home/_navbar.html.erb (59.2ms)
2018-02-15T16:44:57 1d8d4696 [app] [I]   Rendered layouts/base.html.erb (121.9ms)
2018-02-15T16:44:57 1d8d4696 [app] [I] Completed 200 OK in 356ms (Views: 297.7ms | ActiveRecord: 26.1ms)
2018-02-15T16:44:58 c6fa63df [app] [I] Started GET "/notification_recipients" for 172.23.10.4 at 2018-02-15 16:44:58 +0100
2018-02-15T16:44:58 c6fa63df [app] [I] Processing by NotificationRecipientsController#index as JSON
2018-02-15T16:44:58 c6fa63df [app] [I] Current user: admin (administrator)
2018-02-15T16:44:58 c6fa63df [app] [D] Setting current user thread-local variable to admin
2018-02-15T16:44:58 c6fa63df [app] [D] Setting current location thread-local variable to none
2018-02-15T16:44:58 c6fa63df [app] [D] Setting current organization thread-local variable to none
2018-02-15T16:44:58 c6fa63df [notifications] [D] Cache Hit: notification, reading cache for notification-4
2018-02-15T16:44:58 c6fa63df [app] [D] Body: {"notifications":[{"id":14,"seen":false,"level":"warning","text":"foreman.mydomain.lab has no owner set","created_at":"2018-02-15T14:26:48.127Z","group":"Hosts","actions":{"links":[{"href":"/hosts/foreman.mydomain.lab/edit","title":"Update host"}]}},{"id":13,"seen":false,"level":"info","text":"foreman.mydomain.lab has been deleted successfully","created_at":"2018-02-15T14:22:25.515Z","group":"Hosts","actions":{}},{"id":12,"seen":false,"level":"info","text":"Foreman Community Newsletter - December 2017","created_at":"2018-02-15T02:33:12.199Z","group":"Community","actions":{"links":[{"href":"htp://theforeman.org/2017/12/foreman-community-newsletter-december-2017.html","title":"Open","external":true}]}},{"id":8,"seen":false,"level":"info","text":"Security of Foreman’s templating endpoint","created_at":"2018-02-15T02:33:12.169Z","group":"Community","actions":{"links":[{"href":"htp://theforeman.org/2018/01/templating-security.html","title":"Open","external":true}]}},{"id":4,"seen":false,"level":"info","text":"Foreman Community Newsletter - January 2018","created_at":"2018-02-15T02:33:12.063Z","group":"Community","actions":{"links":[{"href":"htp://theforeman.org/2018/01/foreman-community-newsletter-january-2018.html","title":"Open","external":true}]}}]}
2018-02-15T16:44:58 c6fa63df [app] [I] Completed 200 OK in 6ms (Views: 0.1ms | ActiveRecord: 0.4ms)
2018-02-15T16:44:59 eaa7dc27 [app] [I] Started GET "/job_invocations/37?hosts_needs_refresh=&host_ids_needing_name_update%5B%5D=6&host_ids_needing_status_update%5B%5D=6&_=1518709497980" for 172.23.10.4 at 2018-02-15 16:44:59 +0100
2018-02-15T16:44:59 eaa7dc27 [app] [I] Processing by JobInvocationsController#show as JS
2018-02-15T16:44:59 eaa7dc27 [app] [I]   Parameters: {"hosts_needs_refresh"=>"", "host_ids_needing_name_update"=>["6"], "host_ids_needing_status_update"=>["6"], "_"=>"1518709497980", "id"=>"37"}
2018-02-15T16:44:59 eaa7dc27 [app] [I] Current user: admin (administrator)
2018-02-15T16:44:59 eaa7dc27 [app] [D] Setting current user thread-local variable to admin
2018-02-15T16:44:59 eaa7dc27 [app] [D] Setting current location thread-local variable to none
2018-02-15T16:44:59 eaa7dc27 [app] [D] Setting current organization thread-local variable to none
2018-02-15T16:44:59 eaa7dc27 [app] [I]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/show.js.erb
2018-02-15T16:44:59 eaa7dc27 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_host_name_td.html.erb (1.4ms)
2018-02-15T16:44:59 eaa7dc27 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_host_status_td.html.erb (1.2ms)
2018-02-15T16:44:59 eaa7dc27 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/show.js.erb (52.7ms)
2018-02-15T16:44:59 eaa7dc27 [app] [I] Completed 200 OK in 89ms (Views: 55.9ms | ActiveRecord: 5.6ms)
2018-02-15T16:45:00 50a97bc8 [app] [I] Started GET "/job_invocations/37?hosts_needs_refresh=&host_ids_needing_status_update%5B%5D=6&_=1518709497981" for 172.23.10.4 at 2018-02-15 16:45:00 +0100
2018-02-15T16:45:00 50a97bc8 [app] [I] Processing by JobInvocationsController#show as JS
2018-02-15T16:45:00 50a97bc8 [app] [I]   Parameters: {"hosts_needs_refresh"=>"", "host_ids_needing_status_update"=>["6"], "_"=>"1518709497981", "id"=>"37"}
2018-02-15T16:45:00 50a97bc8 [app] [I] Current user: admin (administrator)
2018-02-15T16:45:00 50a97bc8 [app] [D] Setting current user thread-local variable to admin
2018-02-15T16:45:00 50a97bc8 [app] [D] Setting current location thread-local variable to none
2018-02-15T16:45:00 50a97bc8 [app] [D] Setting current organization thread-local variable to none
2018-02-15T16:45:00 50a97bc8 [app] [I]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/show.js.erb
2018-02-15T16:45:00 50a97bc8 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/_host_status_td.html.erb (0.9ms)
2018-02-15T16:45:00 50a97bc8 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/app/views/job_invocations/show.js.erb (37.6ms)
2018-02-15T16:45:00 50a97bc8 [app] [I] Completed 200 OK in 70ms (Views: 34.8ms | ActiveRecord: 8.2ms)
2018-02-15T16:45:01 77752f0c [app] [I] Started POST "/foreman_tasks/api/tasks/callback" for 172.25.0.13 at 2018-02-15 16:45:01 +0100
2018-02-15T16:45:01 77752f0c [app] [I] Processing by ForemanTasks::Api::TasksController#callback as */*
2018-02-15T16:45:01 77752f0c [app] [I]   Parameters: {"callback"=>{"task_id"=>"940cad10-dbd5-4d11-9e54-c9b770826f9e", "step_id"=>3}, "data"=>{"result"=>[{"output_type"=>"stdout", "output"=>"\r\nPLAY [foreman.mydomain.lab] ******************************************************\r\n\r\nTASK [Gathering Facts] *********************************************************\r\nfatal: [foreman.mydomain.lab]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Failed to connect to the host via ssh: write: Broken pipe\\r\\n\", \"unreachable\": true}\r\n\tto retry, use: --limit @/tmp/foreman-playbook-53b04182-fa9a-419a-85aa-360b79260b02.retry\r\n\r\nPLAY RECAP *********************************************************************\r\nforeman.mydomain.lab         : ok=0    changed=0    unreachable=1    failed=0   \r\n\r\n [WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin\r\n(<ansible.plugins.callback.foreman.CallbackModule object at 0x17e3190>):\r\n('Connection aborted.', error(111, 'Connection refused'))\r\n", "timestamp"=>1518709500.5428278}], "runner_id"=>"08bc38c6-9627-4f23-aa60-4e0318686330", "exit_status"=>4}, "task"=>{}}
2018-02-15T16:45:01 77752f0c [app] [D] Examining client certificate to extract dn and sans
2018-02-15T16:45:01 77752f0c [app] [D] Client sent certificate with subject 'foreman.mydomain.lab' and subject alt names '["foreman.mydomain.lab"]'
2018-02-15T16:45:01 77752f0c [app] [D] Verifying request from ["foreman.mydomain.lab"] against ["foreman.mydomain.lab", "foreman.mydomain.lab"]
2018-02-15T16:45:01 77752f0c [app] [I] Current user: foreman_api_admin (administrator)
2018-02-15T16:45:01 77752f0c [app] [D] Setting current user thread-local variable to foreman_api_admin
2018-02-15T16:45:01 77752f0c [app] [D] Body: {"message":"processing"}
2018-02-15T16:45:01 77752f0c [app] [I] Completed 200 OK in 67ms (Views: 0.3ms | ActiveRecord: 17.9ms)
#2

From your logs, it appears that the host you’re trying to configure with Ansible does not let you SSH into it. Have you configured ~/.ssh/authorized_keys in the target host so that it accepts the key of your host?

http://docs.ansible.com/ansible/latest/intro_getting_started.html#remote-connection-information for more info, let us know if you have any other doubts!

#3

Also, if you go to the “Hosts” tab, you can click on the host to see the actual output from Ansible. That should be more helpful than just looking in the logs. I agree it’s not quite intuitive, we’re working on a redesign of that page that should go out soonish https://github.com/theforeman/foreman_remote_execution/pull/130

#4

Thanks for this answer, but i try to configure my localhost.
ssh/authorized_keys is configured for root with user foreman, foreman-proxy, root et s_ansible (user configured in administer -> settings -> ansible) because I have a little trouble understanding who does what

The user configured in “administer” -> “settings” -> “ansible” is the one with which foreman executes the playbook?
I have configured a ipa user s_ansible and password in this location :

conf
conf.PNG1911×509 62.7 KB

This user (s_ansible) is sudoer with no password and can connect with ssh

s_ansible
[s_ansible@foreman ~]$ ssh s_ansible@foreman.mydomain.lab
Last login: Thu Feb 15 21:19:25 2018 from foreman.mydomain.lab
[s_ansible@foreman ~]$

i have to use sudo to run the callback with this user (s_ansible) or i have this warning :

[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.foreman.CallbackModule object at 0x2b51c90>):
[Errno 13] Permission denied

Output in “Hosts” tab :

1:
PLAY [foreman.mydomain.lab] ******************************************************
   2:
   3:
TASK [Gathering Facts] *********************************************************
   4:
fatal: [foreman.mydomain.lab]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh_exchange_identification: Connection closed by remote host\r\n", "unreachable": true}
   5:
to retry, use: --limit @/tmp/foreman-playbook-278fa212-7838-468e-bcab-67be64dd65be.retry
   6:
   7:
PLAY RECAP *********************************************************************
   8:
foreman.mydomain.lab : ok=0 changed=0 unreachable=1 failed=0
   9:
  10:
[WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin
  11:
(<ansible.plugins.callback.foreman.CallbackModule object at 0x1a1c190>):
  12:
('Connection aborted.', error(111, 'Connection refused'))
  13:
Exit status: 4
#5

If it can help, i have configurer foreman like this :

foreman-installer --scenario katello --enable-foreman-compute-vmware --enable-foreman-plugin-ansible --certs-server-cert “/root/foreman.mydomain.lab.crt” --certs-server-cert-req “/root/foreman.mydomain.lab.csr” --certs-server-key “/root/foreman.mydomain.lab.key” --certs-server-ca-cert “/root/idm.mydomain.lab.crt” --enable-foreman-proxy-plugin-ansible --foreman-proxy-dhcp true --foreman-proxy-dns=true --enable-foreman-proxy-plugin-ansible --no-enable-puppet --puppet-agent false --puppet-server false --foreman-proxy-puppet false --foreman-proxy-dhcp-interface ens192 --foreman-proxy-dhcp-gateway 172.25.0.13 --foreman-proxy-dhcp-range “172.25.0.30 172.25.0.250” --foreman-proxy-dhcp-nameservers “foreman.mydomain.lab” --foreman-proxy-dns-interface ens192 --foreman-proxy-dns-zone mydomain.lab --foreman-proxy-dns-reverse 0.25.172.in-addr.arpa --foreman-proxy-dns-forwarders 172.25.0.10

#6

if i run a tcpdump on port 22 (on server and host) i see nothing happened.

#7

I found the problem, when ipa client is disabled it works !!!
But now i search the solution with ipa enable !!

#8

Did you ever find a solution to this problem. I have the same behavior as you with 1.18 and also in a newly installed 1.19. I never see the ssh attempt using tcpdump. I tried with both an ipa-client enabled host and non-ipa-client enable host, neither host worked.

Thanks.

#9

Sorry, I don’t have idea why with IPA enabled it wouldn’t work but to clarify questions from previous comments.

Administer -> Settings -> Ansible -> Username and Password is no longer available. You could still configure that via host parameters, the params would be called ansible_username and ansible_password. These would be passed to ansible-playbook command and stand for SSH connection user. If they are not set, it defaults to what’s set under Administer -> Settings -> RemoteExecution -> SSH User, which is root by default.

There’s also effective user, which is something different. When SSH connection is established, ansible can use sudo to change effective user under which the run performes (also known as become user)

The ansible-playbook command itself (that uses connection user for SSH connection and runs the playbook on target host under effective user) is executed on smart proxy (or Foreman side) which runs under “foreman-proxy” user (“foreman” in case of pure Foreman).

So there are three different users in the whole workflow. Make sure your foreman-proxy user has access to SSH key (if used), it can connect to target host (sudo -u foreman-proxy ssh $connection_user@$target) and that it can sudo -u $effective whoami user on the target machine.

Hope that helps debugging

#10

I migrated to version 1.19

before, the roles execute correctly ,but now it’s KO :frowning:

Could you help me please

#11

Foreman is pretty complex software project with many components and
extensive configuration. Always try to describe best your specific case
(OS, version, ruby and foreman version, database, specific configuration)
and most importantly the error message you see.

#13

Hello,

Thanks for your feedback

the Foreman version is : 1.19 , ansible plugin v 2.0.3 , ansible-2.7.2-1.el7.noarch

Before migration the roles execute correctly , the roles have been launched with the user “foreman” but after upgarde they are launched with user “foreman-proxy” … the problem is that i have ssh keys on the client servers with user “foreman” … I copied the ssh conf from /usr/share/foreman to /usr/share/foreman-proxy and I changed conf on a client server , ssh tests are OK from the terminal the roles execution are KO from web interface error :

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************

fatal: [my_client_server]: UNREACHABLE! => {“changed”: false, “msg”: “Failed to connect to the host via ssh: no such identity: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy: No such file or directory\r\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n”, “unreachable”: true}

to retry, use: --limit @/tmp/foreman-playbook-104404f5-a857-456c-bf99-3a31d4b9f9a8.retry

PLAY RECAP *********************************************************************

my_client_server : ok=0 changed=0 unreachable=1 failed=0

Exit status: 4