Foreman 2.4 - ansible plugin - Plugin uses ansible-playbook instead of ansible-runner (Ubuntu 18.04)

Hello there !

I’m trying to install a fresh Foreman 2.4 with Ansible plugin. Install and configuration were successful but the Ansible plugin seems to be using the ansible-playbook implementation instead of the ansible-runner implementation.

This is a bit disturbing as my another production setup with Foreman 2.1 is correctly using ansible-runner (and both environments have been deployed with the exact same Ansible playbook – I just upgraded the packages versions).

I started to debug by myself but got stuck at some point. I hope that someone could help be understand what’s going wrong !

Problem:
When launching a new job using the Ansible provider (such as “Run command - Ansible default”) through the Foreman UI, on a fresh 2.4 install, the Smart-Proxy executes the job through the “ansible-playbook” instead of “ansible-runner” (as it does on a production setup with Foreman 2.1 on Ubuntu 18.04).

Analyzing the logs shows that the proxy respond a 404 to the initial Foreman request with ‘ansible-runner’ operation (in the proxy.log file) :

Finished POST /dynflow/tasks/launch with 404

Expected outcome: The job should be run using ansible-runner instead of ansible-playbook.

Foreman and Proxy versions: 2.4

Foreman and Proxy plugin versions:
dynflow: 1.4.6-1
smart-proxy-dynflow: 0.3.0-1
smart-proxy-dynflow-core: 0.3.3-1
foreman-tasks-core: 0.3.4-1
foreman-remote-execution-core: 1.4.0-1
foreman-ansible-core: 4.0.0-2
smart-proxy-ansible: 3.0.0-4

Distribution and version: Ubuntu 18.04

Other relevant data:

So I’m not a ruby-expert but learned a bit of ruby thanks to Foreman, in order to debug miscellaneous things.

So far, I’ve found the following:

  • The initial request from Foreman can be seen in the logs:
    Started POST /dynflow/tasks/launch
  • The response is as follows:
    Finished POST /dynflow/tasks/launch with 404 (5.39 ms)
    Nothing about dynflow execution plans so far
  • Foreman seems to try a new request after that:
    Started POST /dynflow/tasks/
  • After a bit of work (dynflow execution plan messages), the reponse is:
    Finished POST /dynflow/tasks/

So it seems the second call is a fallback in case the first did fail, and that this second call actually uses the (old ?) ansible-playbook code from the plugin.

After digging through the code, I’ve found that the following lines are responsible for the “404” error:
/usr/lib/ruby/vendor_ruby/smart_proxy_dynflow_core/api.rb

    def launcher_class(params)
      operation = params.fetch('operation')
      Log.instance.info(params)
      if TaskLauncherRegistry.key?(operation)
        TaskLauncherRegistry.fetch(operation)
      else
        halt 404, MultiJson.dump(:error => "Unknown operation '#{operation}' requested.")
      end
    end

(I added the Log line in order to ensure this comes at the same exact moment as the 404 error, and in order to see the request. For the record, I can see a "operation"=>"ansible-runner" in the request.)

So I then got the the foreman-ansible-core code and found out where the plugin registers itself to the dynflow core:
/usr/lib/ruby/vendor_ruby/foreman_ansible_core.rb:

  if defined?(SmartProxyDynflowCore)
    puts "SmartProxyDynflowCore is defined"
    SmartProxyDynflowCore::TaskLauncherRegistry.register('ansible-runner',
                                                         ForemanAnsibleCore::TaskLauncher::AnsibleRunner)
    SmartProxyDynflowCore::TaskLauncherRegistry.register('ansible-playbook',
                                                         ForemanAnsibleCore::TaskLauncher::Playbook)
  else
    puts "SmartProxyDynflowCore is undefined"
  end

(I’ve added the “puts” line in order to check whether the ansible-runner was registered, and I indeed got a “SmartProxyDynflowCore is undefined” message in the systemd journal).

So now I’m stuck. This is a test setup so I can break anything (I’ve made my own Ansible roles/playbooks to deploy Foreman through packages and wanted to test the upgrade from 2.1 to 2.4).

I am not enough experimented with ruby in order to be able to understand how the SmartProxyDynflowCore gets defined. I think I’ve installed all the required packages (as shown in the Foreman/Smart-Proxy plugins versions section).

Note that with Ubuntu, I’m using the external_core: false as trying to configure the external core does results in another error “uninitialized constant ForemanAnsibleCore” on Foreman 2.4.

Okay so to provide another bunch of diagnosis, I’ve tried the same things with:

  • Foreman 2.1 <== Uses ‘ansible-runner’
  • Foreman 2.2 <== Uses ‘ansible-playbook’
  • Foreman 2.3 <== Uses ‘ansible-playbook’
  • Foreman 2.4 <== Uses ‘ansible-playbook’
    (all on Ubuntu 18.04)

I tried to do something similar to smart_proxy_remote_execution_ssh/plugin.rb with this patch:

--- foreman_ansible_core.rb	2021-05-21 15:41:03.701877324 +0200
+++ /tmp/foreman_ansible_core.rb	2021-05-21 15:40:55.297820219 +0200
@@ -14,6 +14,12 @@
                               :working_dir => nil)
 
   if ForemanTasksCore.dynflow_present?
+    begin
+      require 'foreman_tasks_core/runner'
+    rescue LoadError # rubocop:disable Lint/HandleExceptions
+      # Dynflow core is not available in the proxy, will be handled
+      # by standalone Dynflow core
+    end
     require 'foreman_ansible_core/runner/playbook'
     require 'foreman_ansible_core/runner/ansible_runner'
     require 'foreman_ansible_core/actions'

And it appears to work (Foreman-Proxy now uses ansible-runner as expected) ! So maybe it is just a dependency ordering error ?
I have no idea whether this patch is the right thing to do but feel free to use it if it does.

For the record, below are the setup with Foreman 2.1 (which works natively with ‘ansible-runner’), and the setup with Foreman 2.2 (which seems broken and uses ‘ansible-playbook’):

Foreman 2.1 setup:

  • Foreman and Proxy versions: 2.1
  • Foreman and Proxy plugin versions:
    • dynflow: 1.4.7-1
    • smart-proxy-dynflow: 0.2.4-1
    • smart-proxy-dynflow-core: 0.2.5-1
    • foreman-tasks-core: 0.3.4-1
    • foreman-remote-execution-core: 1.3.1-1
    • foreman-ansible-core: 3.0.3-1
    • smart-proxy-ansible: 3.0.0-2
  • Distribution and version: Ubuntu 18.04

Foreman 2.2 setup:

  • Foreman and Proxy versions: 2.2
  • Foreman and Proxy plugin versions: (newer packages indicated with <==)
    • dynflow: 1.4.7-1
    • smart-proxy-dynflow: 0.2.4-1
    • smart-proxy-dynflow-core: 0.2.6-1 <==
    • foreman-tasks-core: 0.3.4-1
    • foreman-remote-execution-core: 1.3.1-1
    • foreman-ansible-core: 4.0.0-2
    • smart-proxy-ansible: 3.0.0-4 <==
  • Distribution and version: Ubuntu 18.04

Feel free to ask if you have any additional questions, I may help to provide some additional data.
I’m keeping this as “open” since this may be a dirty patch, and I don’t know what is the correct thing to do.

Okay sorry for the triple post, I took to long to edit, but the above patch is incorrect (I did a mistake when generating it).

Here is the correct patch:

--- foreman_ansible_core.rb	2021-05-21 16:20:26.113846300 +0200
+++ /tmp/foreman_ansible_core.rb	2021-05-21 16:20:05.153703878 +0200
@@ -14,6 +14,12 @@
                               :working_dir => nil)
 
   if ForemanTasksCore.dynflow_present?
+    begin
+      require 'smart_proxy_dynflow_core'
+    rescue LoadError # rubocop:disable Lint/HandleExceptions
+      # Dynflow core is not available in the proxy, will be handled
+      # by standalone Dynflow core
+    end
     require 'foreman_tasks_core/runner'
     require 'foreman_ansible_core/runner/playbook'
     require 'foreman_ansible_core/runner/ansible_runner'

I think I’ve hit this. It was tracked as Bug #32209: REX core tries to register task launchers too early - Foreman Remote Execution - Foreman and should be fixed in foreman-remote-execution-core-1.4.1.

Hello ! Thanks for the link, couldn’t find anything related.

Great so I do think my actual problem is that latest Foreman REX version available is 1.4.0 on Ubuntu 18.04 (Index of /pool/bionic/2.4/r/ruby-foreman-remote-execution-core).

Should I contact someone in order to request the new package version for this platform ?

That person would most likely be me, so consider it done :slight_smile:

1 Like

Perfect, thanks for everything Aruzicka !