Incorrect output from job templates runs (Ansible jobs)

Problem:
Unfortunately, the output of a job template run (Ansible) is still incorrect or confusing. I opened a ticket about this some time ago; a fix was developed, but unfortunately it isn’t working.
When a job template is run against multiple servers, the output is fine for some servers, whilst for others all or part of the task results are missing.

I have the following short playbook:

---

- hosts: all
  become: true
  gather_facts: true

  tasks:

  - name: Preload patches for Rocky (Red Hat)
    when: ansible_facts.distribution_file_variety == 'RedHat'
    block:
      - name: Load patches for Rocky (Red Hat)
        ansible.builtin.dnf:
          name: "*"
          state: latest
          update_only: true
          download_only: true
          update_cache: true 
        changed_when: false
  
  - name: Preload patches for SuSE Linux Enterprise (SLES)
    when: ansible_facts.distribution_file_variety == 'SUSE'
    block:
      - name: Clear zypper cache
        ansible.builtin.shell: zypper clean --all
        environment:
          ZYPP_LOCK_TIMEOUT: 20
        changed_when: false

      - name: Load patches for SusE
        ansible.builtin.shell: zypper --non-interactive update --download-only
        environment:
          ZYPP_LOCK_TIMEOUT: 20
        changed_when: false

      - name: Perform dry run
        community.general.zypper:
          name: "*"
          state: latest
          update_cache: false
          extra_args: "--dry-run"        
        environment:
          ZYPP_LOCK_TIMEOUT: 20
        register: zypper_dry_run
        changed_when: false

...

I run this against several hosts – in my case, twelve. Here I’m showing the output from three selected hosts:

Host A:

PLAY [all] *********************************************************************
ok: [dnsa.dns.shared.ogit.cloud]
ok: [dnsa.dns.shared.ogit.cloud]
dnsa.dns.shared.ogit.cloud : ok=2    changed=0    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0
PLAY RECAP *********************************************************************
Exit status: 0

Host B:

PLAY [all] *********************************************************************
ok: [hanah10a.dev.erp.ogit.cloud]
ok: [hanah10a.dev.erp.ogit.cloud]
TASK [Load patches for SusE] ***************************************************
ok: [hanah10a.dev.erp.ogit.cloud]
ok: [hanah10a.dev.erp.ogit.cloud]
hanah10a.dev.erp.ogit.cloud : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
PLAY RECAP *********************************************************************
Exit status: 0

Host C:

PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [hanah11a.snd.erp.ogit.cloud]
TASK [Clear zypper cache] *****************************************************
ok: [hanah11a.snd.erp.ogit.cloud]
ok: [hanah11a.snd.erp.ogit.cloud]
TASK [Perform dry run] *********************************************************
ok: [hanah11a.snd.erp.ogit.cloud]
hanah11a.snd.erp.ogit.cloud : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
PLAY RECAP *********************************************************************
Exit status: 0

Expected outcome:
If I just run the job on a single host, the output is fine:

PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [hanah10a.dev.erp.ogit.cloud]

TASK [Clear zypper cache] ******************************************************
ok: [hanah10a.dev.erp.ogit.cloud]

TASK [Load patches for SusE] ***************************************************
ok: [hanah10a.dev.erp.ogit.cloud]

TASK [Perform dry run] *********************************************************
ok: [hanah10a.dev.erp.ogit.cloud]
hanah10a.dev.erp.ogit.cloud : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

PLAY RECAP *********************************************************************
Exit status: 0

I know that several years ago, the way in which job output is passed to Foreman was changed for performance reasons. However, I don’t think this should be at the expense of readability. Would it perhaps be possible to introduce a choice? A slightly longer runtime but more readable output, or a faster runtime but less readable output?

Foreman and Proxy versions:

Foreman: 3.19.1
Foreman Proxy: 3.19.1
Katello: 4.21.1.1

Foreman and Proxy plugin versions:

Distribution and version:

Rocky Linux 9.8 (Blue Onyx)

Other relevant data:

At certain scale, the “slightly longer runtime” turns into “the job overloads the proxy machine so much that it tips over and the job doesn’t run at all on any of the hosts”, use with caution.

The choice is there, albeit slightly hidden. You can In Administer > Settings > Tasks, you can switch Allow proxy batch tasks to false to get the “old” behaviour.

With that being said, there’s Fixes #36353 - Route Ansible task output per host by jakduch · Pull Request #117 · theforeman/smart_proxy_ansible · GitHub which could help with this, so there’s still some hope.

Thank you so much for the solution. As we currently never run more than 50 hosts in parallel I think we will have no problems. Let’s see what the future holds.