Problem:
The display of the output of Ansible jobs is inconsistent if the job is executed on several hosts, sometimes the output of some or several tasks is missing. More information in “Other relevant data”
Expected outcome:
One consistent log per host
Foreman and Proxy versions:
Both 3.10.0
Foreman and Proxy plugin versions:
Installed Packages
ansible-collection-theforeman-foreman-4.0.0-2.el8.noarch
candlepin-4.3.12-1.el8.noarch
candlepin-selinux-4.3.12-1.el8.noarch
foreman-3.10.0-1.el8.noarch
foreman-cli-3.10.0-1.el8.noarch
foreman-debug-3.10.0-1.el8.noarch
foreman-dynflow-sidekiq-3.10.0-1.el8.noarch
foreman-installer-3.10.0-1.el8.noarch
foreman-installer-katello-3.10.0-1.el8.noarch
foreman-obsolete-packages-1.6-1.el8.noarch
foreman-postgresql-3.10.0-1.el8.noarch
foreman-proxy-3.10.0-1.el8.noarch
foreman-redis-3.10.0-1.el8.noarch
foreman-release-3.10.0-1.el8.noarch
foreman-selinux-3.10.0-1.el8.noarch
foreman-service-3.10.0-1.el8.noarch
katello-4.12.0-1.el8.noarch
katello-certs-tools-2.9.0-2.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.12.0-1.el8.noarch
katello-debug-4.12.0-1.el8.noarch
katello-repos-4.12.0-1.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
pulpcore-obsolete-packages-1.0-9.el8.noarch
pulpcore-selinux-2.0.1-1.el8.x86_64
python3.11-pulp-ansible-0.20.2-3.el8.noarch
python3.11-pulp-certguard-1.7.1-2.el8.noarch
python3.11-pulp-cli-0.21.2-5.el8.noarch
python3.11-pulp-container-2.16.4-1.el8.noarch
python3.11-pulp-deb-3.0.1-1.el8.noarch
python3.11-pulp-file-1.15.1-2.el8.noarch
python3.11-pulp-glue-0.21.2-3.el8.noarch
python3.11-pulp-python-3.10.0-3.el8.noarch
python3.11-pulp-rpm-3.23.3-1.el8.noarch
python3.11-pulpcore-3.39.11-1.el8.noarch
rubygem-foreman-tasks-9.1.1-1.fm3_11.el8.noarch
rubygem-foreman_ansible-14.0.0-1.fm3_11.el8.noarch
rubygem-foreman_maintain-1.5.1-1.el8.noarch
rubygem-foreman_remote_execution-13.0.0-1.fm3_11.el8.noarch
rubygem-hammer_cli-3.10.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.10.0-1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.20-1.fm3_10.el8.noarch
rubygem-hammer_cli_katello-1.12.0-0.1.pre.master.20240122184453git9a6ae05.el8.noarch
rubygem-katello-4.12.0-1.el8.noarch
rubygem-pulp_ansible_client-0.20.3-1.el8.noarch
rubygem-pulp_certguard_client-1.6.5-1.el8.noarch
rubygem-pulp_container_client-2.16.4-1.el8.noarch
rubygem-pulp_deb_client-3.0.1-1.el8.noarch
rubygem-pulp_file_client-1.15.1-1.el8.noarch
rubygem-pulp_ostree_client-2.1.3-1.el8.noarch
rubygem-pulp_python_client-3.10.0-1.el8.noarch
rubygem-pulp_rpm_client-3.23.2-1.el8.noarch
rubygem-pulpcore_client-3.39.9-1.el8.noarch
rubygem-smart_proxy_pulp-3.3.0-1.fm3_10.el8.noarch
Distribution and version:
Rocky Linux 8.9 (Green Obsidian)
Other relevant data:
A job calls up a role with several tasks (Update operation system):
main.yml:
tasks file for osupdate
-
include_tasks: “update-{{ ansible_pkg_mgr }}.yml”
-
name: Check on an update task (wait up to 20 minutes to complete)
ansible.builtin.async_status:
jid: “{{ update_sleeper.ansible_job_id }}”
register: job_result
until: job_result.finished
retries: ‘{{ update_timeout }}’
delay: 60 -
name: “Cleanup async status”
ansible.builtin.async_status:
jid: “{{ update_sleeper.ansible_job_id }}”
mode: cleanup -
name: Reboot after patching
ansible.builtin.shell: “sleep 2 && reboot”
async: 1
poll: 0
when: reboot|bool == true -
name: Wait for the reboot to complete
ansible.builtin.wait_for_connection:
connect_timeout: 5
sleep: 5
delay: 30
timeout: 600
when: reboot|bool == true
…
Task for zypper update:
-
ansible.builtin.debug:
msg: “Checking for running update process” -
ansible.builtin.stat:
path: /var/run/zypp.pid
register: pidfile -
ansible.builtin.lineinfile:
path: /var/run/zypp.pid
regexp: ‘\d+’
state: absent
check_mode: yes
register: zypperpid
until: zypperpid.found == 0
retries: 12
delay: 10
failed_when: false
changed_when: false
when: pidfile.stat.exists -
name: “Get the pid of blocking application”
ansible.builtin.shell: cat /var/run/zypp.pid
register: blockpid
changed_when: false
when: pidfile.stat.exists and zypperpid.found == 1 -
name: “Fail if a running zypper task locking the system”
ansible.builtin.fail:
msg: “System management is locked by the application with pid {{ blockpid.stdout }} - Abort”
when: pidfile.stat.exists and zypperpid.found == 1 -
name: Update server (zypper)
community.general.zypper:
name: ‘’
state: latest
extra_args: ‘–auto-agree-with-licenses’
environment:
ACCEPT_EULA: ‘y’
async: '{{ (update_timeout60)|int|abs }}’
poll: 0
register: update_sleeper
…
Task for dnf Update:
-
ansible.builtin.debug:
msg: “Checking for running update process” -
ansible.builtin.shell: ps -ef|grep ‘/usr/bin/dnf’|grep -E ‘update|upgrade’|grep -v grep
retries: 12
delay: 10
register: dnfpid
failed_when: false
until: dnfpid.rc == 1 -
name: “Fail if a running dnf task is blocking the system”
ansible.builtin.fail:
msg: “System management is locked by another dnf update/upgrade process - Abort”
when: dnfpid.rc == 0 -
name: Update server (dnf)
ansible.builtin.dnf:
name: ‘’
state: latest
async: '{{ (update_timeout60)|int|abs }}’
poll: 0
register: update_sleeper
…
The output of the two hosts against which the job was executed is not consistent:
Output for the Rocky Linux Server:
1:
[WARNING]: Callback disabled by environment. Disabling the Foreman callback
2:
plugin.
3:
4:
PLAY [all] *********************************************************************
5:
6:
TASK [Gathering Facts] *********************************************************
7:
ok: [testrocky02.playground.sandbox.ogit.cloud]
8:
9:
TASK [Update Server] ***********************************************************
10:
11:
TASK [ottogroup.runsap.osupdate : include_tasks] *******************************
12:
included: /usr/share/ansible/collections/ansible_collections/ottogroup/runsap/roles/osupdate/tasks/update-dnf.yml for testrocky02.playground.sandbox.ogit.cloud
13:
included: /usr/share/ansible/collections/ansible_collections/ottogroup/runsap/roles/osupdate/tasks/update-zypper.yml for testsap01.playground.sandbox.ogit.cloud
14:
15:
TASK [ottogroup.runsap.osupdate : ansible.builtin.debug] ***********************
16:
ok: [testrocky02.playground.sandbox.ogit.cloud] => {
17:
“msg”: “Checking for running update process”
18:
}
19:
20:
TASK [ottogroup.runsap.osupdate : ansible.builtin.shell] ***********************
21:
changed: [testrocky02.playground.sandbox.ogit.cloud]
22:
23:
TASK [ottogroup.runsap.osupdate : Update server (dnf)] *************************
24:
changed: [testrocky02.playground.sandbox.ogit.cloud]
25:
FAILED - RETRYING: [testrocky02.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (15 retries left).
26:
FAILED - RETRYING: [testrocky02.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (14 retries left).
27:
FAILED - RETRYING: [testrocky02.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (13 retries left).
28:
FAILED - RETRYING: [testrocky02.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (12 retries left).
29:
FAILED - RETRYING: [testrocky02.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (11 retries left).
30:
FAILED - RETRYING: [testrocky02.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (10 retries left).
31:
FAILED - RETRYING: [testrocky02.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (9 retries left).
32:
changed: [testrocky02.playground.sandbox.ogit.cloud]
33:
34:
TASK [ottogroup.runsap.osupdate : Cleanup async status] ************************
35:
ok: [testrocky02.playground.sandbox.ogit.cloud]
36:
changed: [testrocky02.playground.sandbox.ogit.cloud]
37:
38:
TASK [ottogroup.runsap.osupdate : Wait for the reboot to complete] *************
39:
ok: [testrocky02.playground.sandbox.ogit.cloud]
40:
PLAY RECAP *********************************************************************
41:
testrocky02.playground.sandbox.ogit.cloud : ok=9 changed=4 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
42:
Exit status: 0
Output for SuSE Server:
1:
[WARNING]: Callback disabled by environment. Disabling the Foreman callback
2:
plugin.
3:
4:
PLAY [all] *********************************************************************
5:
ok: [testsap01.playground.sandbox.ogit.cloud]
6:
included: /usr/share/ansible/collections/ansible_collections/ottogroup/runsap/roles/osupdate/tasks/update-dnf.yml for testrocky02.playground.sandbox.ogit.cloud
7:
included: /usr/share/ansible/collections/ansible_collections/ottogroup/runsap/roles/osupdate/tasks/update-zypper.yml for testsap01.playground.sandbox.ogit.cloud
8:
9:
TASK [ottogroup.runsap.osupdate : ansible.builtin.debug] ***********************
10:
ok: [testsap01.playground.sandbox.ogit.cloud] => {
11:
“msg”: “Checking for running update process”
12:
}
13:
14:
TASK [ottogroup.runsap.osupdate : ansible.builtin.stat] ************************
15:
ok: [testsap01.playground.sandbox.ogit.cloud]
16:
17:
TASK [ottogroup.runsap.osupdate : ansible.builtin.lineinfile] ******************
18:
ok: [testsap01.playground.sandbox.ogit.cloud]
19:
20:
TASK [ottogroup.runsap.osupdate : Update server (zypper)] **********************
21:
changed: [testsap01.playground.sandbox.ogit.cloud]
22:
FAILED - RETRYING: [testsap01.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (15 retries left).
23:
FAILED - RETRYING: [testsap01.playground.sandbox.ogit.cloud]: Check on an update task (wait up to 20 minutes to complete) (14 retries left).
24:
25:
TASK [ottogroup.runsap.osupdate : Check on an update task (wait up to 20 minutes to complete)] ***
26:
changed: [testsap01.playground.sandbox.ogit.cloud]
27:
ok: [testsap01.playground.sandbox.ogit.cloud]
28:
29:
TASK [ottogroup.runsap.osupdate : Reboot after patching] ***********************
30:
changed: [testsap01.playground.sandbox.ogit.cloud]
31:
ok: [testsap01.playground.sandbox.ogit.cloud]
32:
PLAY RECAP *********************************************************************
33:
testsap01.playground.sandbox.ogit.cloud : ok=10 changed=3 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
34:
Exit status: 0
As you can see, the output is pretty wildly distributed between the hosts. Incidentally, the problem affects all jobs that call roles.
