Problem:
When running an Ansible playbook job invocation for a host, the live output from Ansible no longer appears on the job invocation status page. Instead, an empty black box is displayed until the job finishes, after which the output of the entire run appears all at once.
Expected outcome:
The output from running the Ansible play should be continuously updated on the job invocation page throughout the entire lifecycle of the job.
Foreman Package versions:
ansible-collection-theforeman-foreman-2.1.2-1.el8.noarch
foreman-2.5.2-1.el8.noarch
foreman-cli-2.5.2-1.el8.noarch
foreman-debug-2.5.2-1.el8.noarch
foreman-dynflow-sidekiq-2.5.2-1.el8.noarch
foreman-installer-2.5.2-3.el8.noarch
foreman-libvirt-2.5.2-1.el8.noarch
foreman-ovirt-2.5.2-1.el8.noarch
foreman-postgresql-2.5.2-1.el8.noarch
foreman-proxy-2.5.2-1.el8.noarch
foreman-release-2.5.2-1.el8.noarch
foreman-selinux-2.5.2-1.el8.noarch
foreman-service-2.5.2-1.el8.noarch
foreman-vmware-2.5.2-1.el8.noarch
rubygem-foreman_ansible-6.3.1-1.fm2_5.el8.noarch
rubygem-foreman_ansible_core-4.1.1-1.fm2_5.el8.noarch
rubygem-foreman_bootdisk-17.1.0-1.fm2_5.el8.noarch
rubygem-foreman_default_hostgroup-6.0.0-1.fm2_5.el8.noarch
rubygem-foreman_discovery-17.0.1-1.fm2_5.el8.noarch
rubygem-foreman_expire_hosts-7.0.4-2.fm2_5.el8.noarch
rubygem-foreman_hooks-0.3.17-2.fm2_5.el8.noarch
rubygem-foreman_maintain-0.8.6-1.el8.noarch
rubygem-foreman_remote_execution-4.5.1-1.fm2_5.el8.noarch
rubygem-foreman_remote_execution-cockpit-4.5.1-1.fm2_5.el8.noarch
rubygem-foreman_remote_execution_core-1.4.4-1.el8.noarch
rubygem-foreman_setup-7.0.0-3.fm2_5.el8.noarch
rubygem-foreman_snapshot_management-2.0.1-1.fm2_5.el8.noarch
rubygem-foreman-tasks-4.1.2-1.fm2_5.el8.noarch
rubygem-foreman-tasks-core-0.3.5-1.fm2_5.el8.noarch
rubygem-foreman_templates-9.1.0-1.fm2_5.el8.noarch
rubygem-hammer_cli_foreman-2.5.1-1.el8.noarch
rubygem-hammer_cli_foreman_discovery-1.0.2-2.fm2_5.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.1-2.fm2_5.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.15-2.fm2_5.el8.noarch
rubygem-hammer_cli_foreman_templates-0.2.0-2.fm2_5.el8.noarch
rubygem-smart_proxy_ansible-3.1.0-1.fm2_5.el8.noarch
rubygem-smart_proxy_dhcp_infoblox-0.0.16-6.el8.noarch
rubygem-smart_proxy_discovery-1.0.5-7.fm2_5.el8.noarch
rubygem-smart_proxy_dynflow-0.3.0-3.fm2_5.el8.noarch
rubygem-smart_proxy_dynflow_core-0.3.3-1.fm2_5.el8.noarch
rubygem-smart_proxy_remote_execution_ssh-0.3.2-1.fm2_5.el8.noarch
Distribution and version:
CentOS 8.4
Other relevant data:
The Foreman instance and smart proxy are both located on the same host.
The problem began happening after a recent multi-version upgrade from 2.3. The intermediate upgrade to 2.4 worked without issue, with Ansible jobs displaying properly in 2.4. However, after the subsequent upgrade from 2.4 to 2.5, this problem began occurring.
No errors appear in logs in /var/log/foreman or /var/log/foreman-proxy that point to something going wrong. All requests to /dynflow/tasks/$UUID on the proxy are showing as returning status 200 while the job is running, and all requests to /template_invocations/$JOBID on the Foreman main instance are showing as rendering without any issues.
When looking at the response JS that my browser is getting to the update requests it sends to /template_invocations/$JOBID, it always contains “var lines = ‘’;”, except for the final request, which has a “lines” variable containing the HTML of the full output of the job.
I’m not familiar enough with how Foreman works to know the best place to look to try and debug this, but from reading around it looks like there have been some dynflow changes recently, so I’m wondering if it might be related to that.