**Problem: If I use “Run Ansible Roles” from a host, the ansible callback is called in the playbook and reports are updated but If I run a remote execution job with Job category * Ansible playbook and Job template defined by me (which is collecting facts) the playbook shows no ansible callback being used and thus no reports are updated
*Expected outcome: For every ansible playbook run which by default is gathering facts the ansible callback should be called and reports should be updated for the host.
I have enabled the ansible callback in the /etc/ansible/ansible.cfg file.
**Foreman and Proxy versions:**1.19.0
Foreman and Proxy plugin versions:
oreman-tasks 0.13.4
foreman_ansible 2.2.9
foreman_remote_execution 1.5.6 Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)
Im using /etc/ansible/ansible.cfg ( the problem is that when I run the default ansible role via Run Ansible Roles button or via Schedule Remote Job -> Ansible Playbook -> Ansible Roles -> Ansible default) the callback works without any issues for any other playbook it doesnt.
I`ve even tried to make a ansible.cfg in /etc/foreman-proxy/ansible.cfg but nothing.
I`ve also noticed that /etc/ansible/ansible.cfg is read for every ansilbe run triggered from the proxy.
I have everything configured, the problem is that sometimes they work sometimes they dont depending on the ansible playbook that I use. Still I dont know where the issue is as I`m using gather facts for all my playbooks.
If an ansible playbook works for one time (eg the callback one) it will work for each other calls. Maybe is something wrong on your playbook or you have network problems. Can you check it?
You can check you production.log or use foreman-tail, when an ansible callback is called (in this case properly) you will have a log like this:
2018-11-20T15:21:10 [I|app|] Started POST “/foreman_tasks/api/tasks/callback” for 10.140.16.230 at 2018-11-20 15:21:10 +0000
2018-11-20T15:21:10 [I|app|d8355] Processing by ForemanTasks::Api::TasksController#callback as /
2018-11-20T15:21:10 [I|app|8d0d8] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/app/views/job_invocations/_host_status_td.html.erb (1.3ms)
2018-11-20T15:21:10 [I|app|d8355] Parameters: {“callback”=>{“task_id”=>“8ae0d6ea-85a2-4e0e-a20f-ef5256d1fbc3”, “step_id”=>3}, “data”=>{“result”=>[{“output_type”=>“stdout”, “output”=>"\r\nPLAY [all] *********************************************************************\r\n\r\nTASK [Gathering Facts] *********************************************************\r\n", “timestamp”=>1542727256.1952581}, {“output_type”=>“stdout”, “output”=>“ok: [myserver]\r\n”, “timestamp”=>1542727260.5177782}, {“output_type”=>“stdout”, “output”=>"\r\nTASK [Display all parameters known for the Foreman host] ***********************\r\n", “timestamp”=>1542727268.135399}, {“output_type”=>“stdout”, “output”=>“ok: [myserver] => {\r\n “foreman_params”: {\r\n “remote_execution_connect_by_ip”: false, \r\n “remote_execution_effective_user_method”: “sudo”, \r\n “remote_execution_ssh_keys”: [\r\n “ssh-rsa AAAAB3N… foreman-proxy@satellite”\r\n ], \r\n “remote_execution_ssh_user”: “root”\r\n }\r\n}\r\n\r\nPLAY RECAP *********************************************************************\r\nmyserver : ok=2 changed=0 unreachable=0 failed=0 \r\n\r\n”, “timestamp”=>1542727269.139767}], “runner_id”=>“240f6a78-29d3-4c1d-8f47-ccb3c35ccb0c”, “exit_status”=>0}, “task”=>{}}
Note that we don’t import configuration report for every ansible run triggered from Foreman. Just those that “play roles” assigned to host/hostgroup. For arbitrary runs, the “report” is only archived in job output. We thought storing a report under non regular/non converging ansible run and hence changing host configuration status on based on things like “install package” or “restart httpd”, would be too confusing.
I had a similar problem just now, several ansible callback (30% of my servers) hanged without a reason and all the following “recurring logic” job (every 30 minutes) were unable to complete the run. I checked into Monitor -> Jobs and Tasks and a lot job/task were in waiting and several ansible processes into the server were in hang state
To solve the issue I ran:
Checked that there are no important jobs or steps running.
Cleaned the postgres db of foreman for all running jobs/task
su - postgres
psql foreman
delete from foreman_tasks_tasks where id in(select id from foreman_tasks_tasks where state = ‘stopped’ and result = ‘error’);
delete from foreman_tasks_tasks where id in(select id from foreman_tasks_tasks where state = ‘stopped’ and result = ‘warning’);
delete from foreman_tasks_tasks where id in(select id from foreman_tasks_tasks where state = ‘paused’ and result = ‘error’);
delete from foreman_tasks_tasks where id in(select id from foreman_tasks_tasks where state = ‘running’ and result = ‘warning’);
delete from foreman_tasks_tasks where id in(select id from foreman_tasks_tasks where state = ‘running’ and result = ‘pending’);
Ran a task cleanup
foreman-rake foreman_tasks:cleanup
Restarted foreman/katello
All hanged task/jobs disappear (obviously) and the next “callback call” will work properly.
