Problem running ansible role(s) on host

Problem:
Hi,

I would like to run ansible roles to my hosts, but it hangs.
When I go to the task details, I see it get stuck at 25%.
In “Running steps” I see a status “suspended”, and as input the details regarding the ansible-runner: hostvars, ansible_user, ansible_ssh_private_key_location and so on.

Expected outcome:
ansible role(s) being executed and related logging as result.

Foreman and Proxy versions:
Foreman and proxy 1.24.2
katello 3.14.1
Ansible 2.9.2

Foreman and Proxy plugin versions:
foreman_ansible_core 3.0.2
foreman_ansible 4.0.4
foreman_remote_execution 2.0.8

Distribution and version:
CentOS Linux release 7.7.1908 (Core)

Other relevant data:
Foreman plugins installed:
–enable-foreman-plugin-ansible
–enable-foreman-proxy-plugin-ansible
–enable-foreman-plugin-remote-execution
–enable-foreman-proxy-plugin-remote-execution-ssh

Config-files in /etc/ansible:
egrep -v ‘^#|^$’ /etc/ansible/foreman.ini
[foreman]
url = http://:3000/
user = foreman
password = secret
ssl_verify = True
use_reports_api = True
[report]
[ansible]
group_patterns = ["{app}-{tier}-{color}",
“{app}-{color}”,
“{app}”,
“{tier}”]
group_prefix = foreman_
want_facts = True
want_hostcollections = False
rich_params = False
want_ansible_ssh_host = False
[cache]
path = .
max_age = 60
scan_new_hosts = True

egrep -v ‘^#|^$’ /etc/ansible/ansible.cfg
[defaults]
callback_whitelist = foreman
bin_ansible_callbacks = true
[inventory]
[privilege_escalation]
[paramiko_connection]
[ssh_connection]
[persistent_connection]
[accelerate]
[selinux]
[colors]
[diff]
[callback_foreman]
url = ‘https://’
ssl_cert = /etc/foreman-proxy/ssl_cert.pem
ssl_key = /etc/foreman-proxy/ssl_key.pem
verify_certs = /etc/foreman-proxy/ssl_ca.pem

Log from production.log:

2020-03-06T11:02:21 [I|app|1296a6b3] Started POST “/job_invocations?feature=ansible_run_host&host_ids%5B%5D=14” for 10.16.8.1 at 2020-03-06 11:02:21 +0100
2020-03-06T11:02:21 [I|app|1296a6b3] Processing by JobInvocationsController#create as HTML
2020-03-06T11:02:21 [I|app|1296a6b3] Parameters: {“authenticity_token”=>“67OSIGVKzffHjgCZdyTSDN8QrxoBYULQhilZurV4kZLsyjm+c4YmtHjoUQdjKfsxPKlJg4M/ae/Mn+Kzs0Wi/w==”, “feature”=>“ansible_run_host”, “host_ids”=>[“14”]}
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on job_category Ansible Playbook
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on description Run Ansible roles
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on concurrency_level
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on time_span
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on execution_timeout_interval
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on password [redacted]
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on key_passphrase
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on remote_execution_feature_id 11
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on sudo_password
2020-03-06T11:02:22 [I|bac|1296a6b3] Task {label: Actions::RemoteExecution::RunHostsJob, id: 9b506b43-c626-4382-b9e4-e6ea7048f1f1, execution_plan_id: 03eda303-502b-4b40-9210-24ae4327413b} state changed: planning
2020-03-06T11:02:22 [I|bac|] Task {label: Actions::RemoteExecution::RunHostsJob, id: 9b506b43-c626-4382-b9e4-e6ea7048f1f1, execution_plan_id: 03eda303-502b-4b40-9210-24ae4327413b} state changed: planned
2020-03-06T11:02:22 [I|app|] Redirected to https:///job_invocations/30
2020-03-06T11:02:22 [I|app|] Completed 302 Found in 541ms (ActiveRecord: 127.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Started GET “/job_invocations/30” for 10.16.8.1 at 2020-03-06 11:02:22 +0100
2020-03-06T11:02:22 [I|app|44be7c23] Processing by JobInvocationsController#show as HTML
2020-03-06T11:02:22 [I|app|44be7c23] Parameters: {“id”=>“30”}
2020-03-06T11:02:22 [I|app|44be7c23] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.html.erb within layouts/application
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_card_results.html.erb (0.6ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_card_schedule.html.erb (2.0ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_card_target_hosts.html.erb (3.4ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_name_td.html.erb (1.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_status_td.html.erb (0.9ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_actions_td.html.erb (2.3ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered common/_pagination.html.erb (1.5ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_tab_hosts.html.erb (41.5ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_tab_overview.html.erb (70.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_user_input.html.erb (70.0ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/tab_preview_templates.html.erb (86.6ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.html.erb within layouts/application (198.4ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered layouts/application_content.html.erb (1.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendering layouts/base.html.erb
2020-03-06T11:02:22 [I|app|44be7c23] Rendered layouts/base.html.erb (58.2ms)
2020-03-06T11:02:22 [I|app|44be7c23] Completed 200 OK in 302ms (Views: 242.8ms | ActiveRecord: 28.2ms)
2020-03-06T11:02:23 [I|app|14fb63da] Started GET “/notification_recipients” for 10.16.8.1 at 2020-03-06 11:02:23 +0100
2020-03-06T11:02:23 [I|app|14fb63da] Processing by NotificationRecipientsController#index as JSON
2020-03-06T11:02:23 [I|app|14fb63da] Completed 200 OK in 13ms (Views: 0.2ms | ActiveRecord: 2.0ms)
2020-03-06T11:02:23 [I|app|00e394ed] Started GET “/job_invocations/chart?id=30” for 10.16.8.1 at 2020-03-06 11:02:23 +0100
2020-03-06T11:02:23 [I|app|00e394ed] Processing by JobInvocationsController#chart as JSON
2020-03-06T11:02:23 [I|app|00e394ed] Parameters: {“id”=>“30”}
2020-03-06T11:02:23 [I|app|00e394ed] Completed 200 OK in 60ms (Views: 0.3ms | ActiveRecord: 5.6ms)
2020-03-06T11:02:23 [I|app|799983cb] Started GET "/job_invocations/30?hosts_needs_refresh=&host_ids_needing_name_update%5B%5D=14&host_ids_needing_status_update%5B%5D=14&
=1583488942871" for 10.16.8.1 at 2020-03-06 11:02:23 +0100
2020-03-06T11:02:23 [I|app|799983cb] Processing by JobInvocationsController#show as JS
2020-03-06T11:02:23 [I|app|799983cb] Parameters: {“hosts_needs_refresh”=>"", “host_ids_needing_name_update”=>[“14”], “host_ids_needing_status_update”=>[“14”], "
"=>“1583488942871”, “id”=>“30”}
2020-03-06T11:02:24 [I|app|799983cb] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.js.erb
2020-03-06T11:02:24 [I|app|799983cb] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_name_td.html.erb (1.3ms)
2020-03-06T11:02:24 [I|app|799983cb] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_status_td.html.erb (1.0ms)
2020-03-06T11:02:24 [I|app|799983cb] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.js.erb (84.5ms)

The part “Started GET “/job_invocations/chart?id=30” for 10.16.8.1 at …” constantly repeated.

I’m also confused about the ip-address mentioned in this log.
It is the default gateway assigned to the host configured to one of the additional interfaces.
It is not the ip-range used during the provisioning of the host, we use separate vlans for PXE and productive purpose.

Any help is more then welcome to find out what might be wrong in our setup/configuration.

What version of ansible-runner do you have installed?

My ansible broke when I upgraded to Foreman 1.24.

It turned out the default version that came with my RHEL7.7 was too old. I upgraded to the latest using pip and ansible started working again.

Working version:

# ansible-runner --version
1.4.4

Mine is as well, ansible-runner --version
1.4.4

In that case I don’t know. I just thought I’d mention it since it’s a quick fix. :upside_down_face:

Additional info:
Running a REX command like df -h works as expected.

Noticed following:

I had to upgrade foreman from initial installation from version 1.23 to 1.24 and later to 1.24.1.
However, as foreman_ansible_core was not the correct version in 1.24.1 I had to upgrade to 1.24.2.

But, with ‘rpm -qa | grep ansible’ I now see rubygem-smart_proxy_ansible-3.0.1-1.fm1_23 which is refering to foreman 1.23.
Should it not be upgraded to version 1.24 as well, or is this the correct version?

That’s what I’ve got.

# rpm -qa | grep ansible
tfm-rubygem-foreman_ansible_core-3.0.2-1.fm1_24.el7.noarch
ansible-2.9.4-1.el7ae.noarch
tfm-rubygem-pulp_ansible_client-0.2.0b1.dev0.1560866833-1.el7.noarch
rubygem-smart_proxy_ansible-doc-3.0.1-1.fm1_23.el7.noarch
ansiblerole-foreman_scap_client-0.0.3-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-0.3.2-1.fm1_24.el7.noarch
rubygem-smart_proxy_ansible-3.0.1-1.fm1_23.el7.noarch
tfm-rubygem-foreman_ansible-4.0.4-1.fm1_24.el7.noarch