Ansible job invocations no longer have live dynamic result updates after upgrading to Foreman 2.5

Problem:
When running an Ansible playbook job invocation for a host, the live output from Ansible no longer appears on the job invocation status page. Instead, an empty black box is displayed until the job finishes, after which the output of the entire run appears all at once.

Expected outcome:
The output from running the Ansible play should be continuously updated on the job invocation page throughout the entire lifecycle of the job.

Foreman Package versions:
ansible-collection-theforeman-foreman-2.1.2-1.el8.noarch
foreman-2.5.2-1.el8.noarch
foreman-cli-2.5.2-1.el8.noarch
foreman-debug-2.5.2-1.el8.noarch
foreman-dynflow-sidekiq-2.5.2-1.el8.noarch
foreman-installer-2.5.2-3.el8.noarch
foreman-libvirt-2.5.2-1.el8.noarch
foreman-ovirt-2.5.2-1.el8.noarch
foreman-postgresql-2.5.2-1.el8.noarch
foreman-proxy-2.5.2-1.el8.noarch
foreman-release-2.5.2-1.el8.noarch
foreman-selinux-2.5.2-1.el8.noarch
foreman-service-2.5.2-1.el8.noarch
foreman-vmware-2.5.2-1.el8.noarch
rubygem-foreman_ansible-6.3.1-1.fm2_5.el8.noarch
rubygem-foreman_ansible_core-4.1.1-1.fm2_5.el8.noarch
rubygem-foreman_bootdisk-17.1.0-1.fm2_5.el8.noarch
rubygem-foreman_default_hostgroup-6.0.0-1.fm2_5.el8.noarch
rubygem-foreman_discovery-17.0.1-1.fm2_5.el8.noarch
rubygem-foreman_expire_hosts-7.0.4-2.fm2_5.el8.noarch
rubygem-foreman_hooks-0.3.17-2.fm2_5.el8.noarch
rubygem-foreman_maintain-0.8.6-1.el8.noarch
rubygem-foreman_remote_execution-4.5.1-1.fm2_5.el8.noarch
rubygem-foreman_remote_execution-cockpit-4.5.1-1.fm2_5.el8.noarch
rubygem-foreman_remote_execution_core-1.4.4-1.el8.noarch
rubygem-foreman_setup-7.0.0-3.fm2_5.el8.noarch
rubygem-foreman_snapshot_management-2.0.1-1.fm2_5.el8.noarch
rubygem-foreman-tasks-4.1.2-1.fm2_5.el8.noarch
rubygem-foreman-tasks-core-0.3.5-1.fm2_5.el8.noarch
rubygem-foreman_templates-9.1.0-1.fm2_5.el8.noarch
rubygem-hammer_cli_foreman-2.5.1-1.el8.noarch
rubygem-hammer_cli_foreman_discovery-1.0.2-2.fm2_5.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.1-2.fm2_5.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.15-2.fm2_5.el8.noarch
rubygem-hammer_cli_foreman_templates-0.2.0-2.fm2_5.el8.noarch
rubygem-smart_proxy_ansible-3.1.0-1.fm2_5.el8.noarch
rubygem-smart_proxy_dhcp_infoblox-0.0.16-6.el8.noarch
rubygem-smart_proxy_discovery-1.0.5-7.fm2_5.el8.noarch
rubygem-smart_proxy_dynflow-0.3.0-3.fm2_5.el8.noarch
rubygem-smart_proxy_dynflow_core-0.3.3-1.fm2_5.el8.noarch
rubygem-smart_proxy_remote_execution_ssh-0.3.2-1.fm2_5.el8.noarch

Distribution and version:
CentOS 8.4

Other relevant data:
The Foreman instance and smart proxy are both located on the same host.

The problem began happening after a recent multi-version upgrade from 2.3. The intermediate upgrade to 2.4 worked without issue, with Ansible jobs displaying properly in 2.4. However, after the subsequent upgrade from 2.4 to 2.5, this problem began occurring.

No errors appear in logs in /var/log/foreman or /var/log/foreman-proxy that point to something going wrong. All requests to /dynflow/tasks/$UUID on the proxy are showing as returning status 200 while the job is running, and all requests to /template_invocations/$JOBID on the Foreman main instance are showing as rendering without any issues.

When looking at the response JS that my browser is getting to the update requests it sends to /template_invocations/$JOBID, it always contains “var lines = ‘’;”, except for the final request, which has a “lines” variable containing the HTML of the full output of the job.

I’m not familiar enough with how Foreman works to know the best place to look to try and debug this, but from reading around it looks like there have been some dynflow changes recently, so I’m wondering if it might be related to that.

Hi,
good catch. There were some changes recently, but only a part of them got released and landed in 2.5 repositories. I’ll release the rest later today, it shouldn’t take too long

Fix will land in foreman-tasks-core-0.3.6. It is released on rubygems, packaging PRs are waiting for review.

Thanks for the quick reply, I wish I could have followed up sooner.

Unfortunately it doesn’t look like updating foreman-tasks-core to 0.3.6 fixed this issue; I applied the update about 10 days ago, and I’ve still been experiencing Ansible jobs not returning output until the job is completed. I’ve also installed all other package updates that have come out in that time, and none of them have resolved it either.

Am I missing any additional steps that need to be taken beyond just installing the latest RPMs? I’ve also tried the few cache clearing commands I’m aware of, and doing full restarts of the Foreman services and the host running them, to no effect.

Turns out there was one more thing that needed fixing. PR with a fix is ready so now it is just a matter of time.

1 Like

I’ve installed the latest round of updates, which included rubygem-foreman-tasks 4.1.4-1, and can verify that my issue is now resolved; I’m once again able to see live output of Ansible jobs. Thanks for the help!