Problem:
Hi,
I would like to run ansible roles to my hosts, but it hangs.
When I go to the task details, I see it get stuck at 25%.
In “Running steps” I see a status “suspended”, and as input the details regarding the ansible-runner: hostvars, ansible_user, ansible_ssh_private_key_location and so on.
Expected outcome:
ansible role(s) being executed and related logging as result.
Foreman and Proxy versions:
Foreman and proxy 1.24.2
katello 3.14.1
Ansible 2.9.2
Foreman and Proxy plugin versions:
foreman_ansible_core 3.0.2
foreman_ansible 4.0.4
foreman_remote_execution 2.0.8
Distribution and version:
CentOS Linux release 7.7.1908 (Core)
Other relevant data:
Foreman plugins installed:
–enable-foreman-plugin-ansible
–enable-foreman-proxy-plugin-ansible
–enable-foreman-plugin-remote-execution
–enable-foreman-proxy-plugin-remote-execution-ssh
Config-files in /etc/ansible:
egrep -v ‘^#|^$’ /etc/ansible/foreman.ini
[foreman]
url = http://:3000/
user = foreman
password = secret
ssl_verify = True
use_reports_api = True
[report]
[ansible]
group_patterns = ["{app}-{tier}-{color}",
“{app}-{color}”,
“{app}”,
“{tier}”]
group_prefix = foreman_
want_facts = True
want_hostcollections = False
rich_params = False
want_ansible_ssh_host = False
[cache]
path = .
max_age = 60
scan_new_hosts = True
egrep -v ‘^#|^$’ /etc/ansible/ansible.cfg
[defaults]
callback_whitelist = foreman
bin_ansible_callbacks = true
[inventory]
[privilege_escalation]
[paramiko_connection]
[ssh_connection]
[persistent_connection]
[accelerate]
[selinux]
[colors]
[diff]
[callback_foreman]
url = ‘https://’
ssl_cert = /etc/foreman-proxy/ssl_cert.pem
ssl_key = /etc/foreman-proxy/ssl_key.pem
verify_certs = /etc/foreman-proxy/ssl_ca.pem
Log from production.log:
2020-03-06T11:02:21 [I|app|1296a6b3] Started POST “/job_invocations?feature=ansible_run_host&host_ids%5B%5D=14” for 10.16.8.1 at 2020-03-06 11:02:21 +0100
2020-03-06T11:02:21 [I|app|1296a6b3] Processing by JobInvocationsController#create as HTML
2020-03-06T11:02:21 [I|app|1296a6b3] Parameters: {“authenticity_token”=>“67OSIGVKzffHjgCZdyTSDN8QrxoBYULQhilZurV4kZLsyjm+c4YmtHjoUQdjKfsxPKlJg4M/ae/Mn+Kzs0Wi/w==”, “feature”=>“ansible_run_host”, “host_ids”=>[“14”]}
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on job_category Ansible Playbook
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on description Run Ansible roles
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on concurrency_level
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on time_span
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on execution_timeout_interval
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on password [redacted]
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on key_passphrase
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on remote_execution_feature_id 11
2020-03-06T11:02:21 [I|aud|1296a6b3] JobInvocation (30) create event on sudo_password
2020-03-06T11:02:22 [I|bac|1296a6b3] Task {label: Actions::RemoteExecution::RunHostsJob, id: 9b506b43-c626-4382-b9e4-e6ea7048f1f1, execution_plan_id: 03eda303-502b-4b40-9210-24ae4327413b} state changed: planning
2020-03-06T11:02:22 [I|bac|] Task {label: Actions::RemoteExecution::RunHostsJob, id: 9b506b43-c626-4382-b9e4-e6ea7048f1f1, execution_plan_id: 03eda303-502b-4b40-9210-24ae4327413b} state changed: planned
2020-03-06T11:02:22 [I|app|] Redirected to https:///job_invocations/30
2020-03-06T11:02:22 [I|app|] Completed 302 Found in 541ms (ActiveRecord: 127.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Started GET “/job_invocations/30” for 10.16.8.1 at 2020-03-06 11:02:22 +0100
2020-03-06T11:02:22 [I|app|44be7c23] Processing by JobInvocationsController#show as HTML
2020-03-06T11:02:22 [I|app|44be7c23] Parameters: {“id”=>“30”}
2020-03-06T11:02:22 [I|app|44be7c23] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.html.erb within layouts/application
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_card_results.html.erb (0.6ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_card_schedule.html.erb (2.0ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_card_target_hosts.html.erb (3.4ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_name_td.html.erb (1.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_status_td.html.erb (0.9ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_actions_td.html.erb (2.3ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered common/_pagination.html.erb (1.5ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_tab_hosts.html.erb (41.5ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_tab_overview.html.erb (70.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_user_input.html.erb (70.0ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/tab_preview_templates.html.erb (86.6ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.html.erb within layouts/application (198.4ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendered layouts/application_content.html.erb (1.7ms)
2020-03-06T11:02:22 [I|app|44be7c23] Rendering layouts/base.html.erb
2020-03-06T11:02:22 [I|app|44be7c23] Rendered layouts/base.html.erb (58.2ms)
2020-03-06T11:02:22 [I|app|44be7c23] Completed 200 OK in 302ms (Views: 242.8ms | ActiveRecord: 28.2ms)
2020-03-06T11:02:23 [I|app|14fb63da] Started GET “/notification_recipients” for 10.16.8.1 at 2020-03-06 11:02:23 +0100
2020-03-06T11:02:23 [I|app|14fb63da] Processing by NotificationRecipientsController#index as JSON
2020-03-06T11:02:23 [I|app|14fb63da] Completed 200 OK in 13ms (Views: 0.2ms | ActiveRecord: 2.0ms)
2020-03-06T11:02:23 [I|app|00e394ed] Started GET “/job_invocations/chart?id=30” for 10.16.8.1 at 2020-03-06 11:02:23 +0100
2020-03-06T11:02:23 [I|app|00e394ed] Processing by JobInvocationsController#chart as JSON
2020-03-06T11:02:23 [I|app|00e394ed] Parameters: {“id”=>“30”}
2020-03-06T11:02:23 [I|app|00e394ed] Completed 200 OK in 60ms (Views: 0.3ms | ActiveRecord: 5.6ms)
2020-03-06T11:02:23 [I|app|799983cb] Started GET "/job_invocations/30?hosts_needs_refresh=&host_ids_needing_name_update%5B%5D=14&host_ids_needing_status_update%5B%5D=14&=1583488942871" for 10.16.8.1 at 2020-03-06 11:02:23 +0100
2020-03-06T11:02:23 [I|app|799983cb] Processing by JobInvocationsController#show as JS
2020-03-06T11:02:23 [I|app|799983cb] Parameters: {“hosts_needs_refresh”=>"", “host_ids_needing_name_update”=>[“14”], “host_ids_needing_status_update”=>[“14”], ""=>“1583488942871”, “id”=>“30”}
2020-03-06T11:02:24 [I|app|799983cb] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.js.erb
2020-03-06T11:02:24 [I|app|799983cb] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_name_td.html.erb (1.3ms)
2020-03-06T11:02:24 [I|app|799983cb] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/_host_status_td.html.erb (1.0ms)
2020-03-06T11:02:24 [I|app|799983cb] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/views/job_invocations/show.js.erb (84.5ms)
The part “Started GET “/job_invocations/chart?id=30” for 10.16.8.1 at …” constantly repeated.
I’m also confused about the ip-address mentioned in this log.
It is the default gateway assigned to the host configured to one of the additional interfaces.
It is not the ip-range used during the provisioning of the host, we use separate vlans for PXE and productive purpose.
Any help is more then welcome to find out what might be wrong in our setup/configuration.