Ansible callback doesn't work after 1.22 upgrade

Problem:

I upgraded our foreman+katello from version 1.20.2 to 1.22 (1.20.2->1.21.3->1.22.0) and after the upgrade the ansible callback is not working anymore (also if used as a recurring logics)
All the VM are out of sync or in error state.

Previously the job was running properly since last August 2018.

Expected outcome:

When I run an ansible roles (callback) the execution will work.

Foreman and Proxy versions:

Foreman 1.2.0
Katello 3.12

Foreman and Proxy plugin versions:

Ansible plugin 3.0.2
Ansible 2.8.0/python 2.7.5

Other relevant data:

Our callback job template, who was working on 1.18/1.19/1.20.x, is:

---
- hosts: all
  tasks:
    - name: Display all parameters known for the Foreman host
      debug:
        var: foreman_params
  roles:
<%- if @host.all_ansible_roles.present? -%>
<%=   @host.all_ansible_roles.map { |role| "    - #{role.name.strip}" }.join("\n") %>
<%- end -%>

When I run a “Play Ansible Roles” it failed and the log is:

2019-06-14T15:39:09 [I|app|722580a9] Current user set to admin (admin)
2019-06-14T15:39:09 [I|app|722580a9]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.7.0/app/views/job_invocations/show.js.erb
2019-06-14T15:39:09 [I|app|f8fd5d37] Started POST "/foreman_tasks/api/tasks/callback" for 10.140.16.230 at 2019-06-14 15:39:09 +0100
2019-06-14T15:39:09 [I|app|f8fd5d37] Processing by ForemanTasks::Api::TasksController#callback as */*
2019-06-14T15:39:09 [I|app|f8fd5d37]   Parameters: {"callback"=>{"task_id"=>"fc661d82-591a-4971-ba7e-31523b1902ac", "step_id"=>3}, "data"=>{"result"=>[{"output_type"=>"stdout", "output"=>"/var/tmp/foreman-ssh-cmd-31f1e247-044d-4dad-a8fb-632b9852c8ff/script: line 1: ---: command not found\r\n", "timestamp"=>1560523149.5836625}, {"output_type"=>"stdout", "output"=>"/var/tmp/foreman-ssh-cmd-31f1e247-044d-4dad-a8fb-632b9852c8ff/script: line 2: -: command not found\r\n", "timestamp"=>1560523149.5838606}, {"output_type"=>"stdout", "output"=>"/var/tmp/foreman-ssh-cmd-31f1e247-044d-4dad-a8fb-632b9852c8ff/script: line 3: tasks:: command not found\r\n", "timestamp"=>1560523149.5841339}, {"output_type"=>"stdout", "output"=>"/var/tmp/foreman-ssh-cmd-31f1e247-044d-4dad-a8fb-632b9852c8ff/script: line 4: -: command not found\r\n", "timestamp"=>1560523149.5851808}, {"output_type"=>"stdout", "output"=>"/var/tmp/foreman-ssh-cmd-31f1e247-044d-4dad-a8fb-632b9852c8ff/script: line 5: debug:: command not found\r\n", "timestamp"=>1560523149.5855129}, {"output_type"=>"stdout", "output"=>"/var/tmp/foreman-ssh-cmd-31f1e247-044d-4dad-a8fb-632b9852c8ff/script: line 6: var:: command not found\r\n", "timestamp"=>1560523149.5857708}, {"output_type"=>"stdout", "output"=>"/var/tmp/foreman-ssh-cmd-31f1e247-044d-4dad-a8fb-632b9852c8ff/script: line 7: roles:: command not found\r\n", "timestamp"=>1560523149.5859997}], "runner_id"=>"31f1e247-044d-4dad-a8fb-632b9852c8ff", "exit_status"=>127}, "task"=>{}}
2019-06-14T15:39:09 [I|app|722580a9]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.7.0/app/views/job_invocations/_host_status_td.html.erb (1.2ms)
2019-06-14T15:39:09 [I|app|722580a9]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.7.0/app/views/job_invocations/show.js.erb (64.4ms)
2019-06-14T15:39:09 [I|app|722580a9] Completed 200 OK in 135ms (Views: 57.6ms | ActiveRecord: 26.9ms

It seems foreman or katello is no more able to use that template.

It is not the callback but the job execution that is not working. I already filed the bug Bug #27054: Ansible playbook is executed as shell script via ssh - Ansible - Foreman for it today.

1 Like

Please check that you have tfm-rubygem-foreman_ansible_core installed. This happens when the gem is not available and we then try to interpret the given input as a regular shell script

The package is installed.

$ rpm -qa | grep ansible
tfm-rubygem-foreman_ansible_core-3.0.0-1.fm1_22.el7.noarch
tfm-rubygem-foreman_ansible-3.0.2-1.fm1_22.el7.noarch
ansible-2.8.0-2.el7.noarch
rubygem-smart_proxy_ansible-2.1.2-3.fm1_22.el7.noarch

How should the gem be loaded?

# grep -R ansible /usr/share/foreman-proxy/bundler.d/
/usr/share/foreman-proxy/bundler.d/smart_proxy_ansible.rb:gem 'smart_proxy_ansible'
# grep -R ansible /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/
/opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb:gem 'foreman_ansible_core'
# grep -R ansible /usr/share/foreman/bundler.d/
/usr/share/foreman/bundler.d/foreman_ansible.rb:gem 'foreman_ansible' 

I had the same problem. Downgrading the 3.0 packages to the 2.x packages seems to fix it.

# yum downgrade  tfm-rubygem-foreman_ansible tfm-rubygem-foreman_ansible_core
Resolving Dependencies
--> Running transaction check
---> Package tfm-rubygem-foreman_ansible.noarch 0:2.3.3-2.fm1_22.el7 will be a downgrade
---> Package tfm-rubygem-foreman_ansible.noarch 0:3.0.2-1.fm1_22.el7 will be erased
---> Package tfm-rubygem-foreman_ansible_core.noarch 0:2.2.1-1.fm1_22.el7 will be a downgrade
---> Package tfm-rubygem-foreman_ansible_core.noarch 0:3.0.0-1.fm1_22.el7 will be erased
--> Finished Dependency Resolution

Correction:
Downgrading the 3.0 packages (as above) fixes Remote Execution running as shell, not ansible.

But the “Ansible Roles” tab on the Create Host appears completely blank - so it’s not a complete fix. :neutral_face:

The “Ansible Roles” tab on the host Edit page, and the Configure->Host Groups still has a working “Ansible Roles” tab, so there are workarounds available.

This issue should be resolved once remote execution 1.8.0 lands in the repos.

2 Likes

I created the packages locally and tested after updating them. ansible-playbook mode which is still default runs fine afterwards, ansible-runner fails without output.

Thanks @aruzicka.

I will give the custom packages to the customer and wait for someone from @packaging to do the Koji magic as I can not do this myself for now.

I’ve fixed the infra side so the pushing works again. 1.8.0 should available in 1.22 now.

3 Likes

That seems to have fixed it. Thanks!

Resolving Dependencies
--> Running transaction check
---> Package tfm-rubygem-foreman_ansible.noarch 0:2.3.3-2.fm1_22.el7 will be updated
---> Package tfm-rubygem-foreman_ansible.noarch 0:3.0.2-1.fm1_22.el7 will be an update
---> Package tfm-rubygem-foreman_ansible_core.noarch 0:2.2.1-1.fm1_22.el7 will be updated
---> Package tfm-rubygem-foreman_ansible_core.noarch 0:3.0.0-1.fm1_22.el7 will be an update
---> Package tfm-rubygem-foreman_remote_execution.noarch 0:1.7.0-12.fm1_22.el7 will be updated
---> Package tfm-rubygem-foreman_remote_execution.noarch 0:1.8.0-1.fm1_22.el7 will be an update
---> Package tfm-rubygem-foreman_remote_execution_core.noarch 0:1.1.5-1.el7 will be updated
---> Package tfm-rubygem-foreman_remote_execution_core.noarch 0:1.2.0-1.el7 will be an update
--> Finished Dependency Resolution
1 Like

Thanks at all involved. I marked the bug as resolved.

1 Like

I confirm that, the upgrade from 1.20 (1.21->1.22) works fine.

It seems like the problem still exists on Ubuntu according to a comment in:
https://projects.theforeman.org/issues/27054#change-122775

Can anyone more familiar with the Debian packaging have a look into?

Perhaps important to @tbrisker

1 Like

Looks like indeed rex 1.8 wasn’t released for debians in the 1.21 tree

I don’t believe that’s the same bug. 1.21 is still on ansible 2.3.5 which does not need rex 1.8.