Using sync task for running Ansible plays hangs Foreman

Hello,

I have an issue with triggering synchronous tasks in foreman-tasks. I have
written a plugin that is triggering an Ansible Play as a foreman-tasks'
sync_task (the foreman_ansible plugin itself uses only asynchronous tasks,
but I want the request in my plugin to particularly block until the
triggered Ansible task has completed). During playbook execution, the
Ansible callback plugin reports gathered facts to the Foreman facts API,
which I need later. However, as soon as the callback plugin contacts the
facts API, Foreman starts hanging completely, not answering any requests
any longer. Restarting httpd makes Foreman in production usable again.
After restarting and trying the call again, the entire procedure often
works as expected, which makes it even more incomprehensible for me. The
problem does not happen at all when the Ansible callback plugin is not
configured. In this case the call always blocks and succeeds as expected.
Does anybody have any idea what could be the reason for this behaviour? And
is there any way to get around this problem? I'd be very grateful for any
hints.

Regards

–Gerrit

> Hello,
>
> I have an issue with triggering synchronous tasks in foreman-tasks. I have
> written a plugin that is triggering an Ansible Play as a foreman-tasks'
> sync_task (the foreman_ansible plugin itself uses only asynchronous
> tasks, but I want the request in my plugin to particularly block until the
> triggered Ansible task has completed). During playbook execution, the
> Ansible callback plugin reports gathered facts to the Foreman facts API,
> which I need later. However, as soon as the callback plugin contacts the
> facts API, Foreman starts hanging completely, not answering any requests
> any longer. Restarting httpd makes Foreman in production usable again.
> After restarting and trying the call again, the entire procedure often
> works as expected, which makes it even more incomprehensible for me. The
> problem does not happen at all when the Ansible callback plugin is not
> configured. In this case the call always blocks and succeeds as expected.
> Does anybody have any idea what could be the reason for this behaviour? And
> is there any way to get around this problem? I'd be very grateful for any
> hints.
>
>
can you provide actual API logs? its pretty hard to troubleshoot without
any data.

··· On Mon, Mar 13, 2017 at 12:09 PM, wrote:

Regards

–Gerrit


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It actually just seems like the entire API is blocked when a sync task is
triggered directly in the controller method. Does it have something to do
with concurrency in Rails? Maybe also resources get locked producing a
deadlock? Still would be happy about some suggestions.

··· On Monday, March 13, 2017 at 11:09:45 AM UTC+1, gerrit.sc...@avid.com wrote: > > Hello, > > I have an issue with triggering synchronous tasks in foreman-tasks. I have > written a plugin that is triggering an Ansible Play as a foreman-tasks' > sync_task (the foreman_ansible plugin itself uses only asynchronous > tasks, but I want the request in my plugin to particularly block until the > triggered Ansible task has completed). During playbook execution, the > Ansible callback plugin reports gathered facts to the Foreman facts API, > which I need later. However, as soon as the callback plugin contacts the > facts API, Foreman starts hanging completely, not answering any requests > any longer. Restarting httpd makes Foreman in production usable again. > After restarting and trying the call again, the entire procedure often > works as expected, which makes it even more incomprehensible for me. The > problem does not happen at all when the Ansible callback plugin is not > configured. In this case the call always blocks and succeeds as expected. > Does anybody have any idea what could be the reason for this behaviour? And > is there any way to get around this problem? I'd be very grateful for any > hints. > > Regards > > --Gerrit >

Hey! Yes, I can. This is form the log:

11:30:56 rails.1 | 2017-03-13T11:30:56 522230d8 [app] [D]

11:30:56 rails.1 | 2017-03-13T11:30:56 522230d8 [app] [D]

11:30:56 rails.1 | 2017-03-13T11:30:56 522230d8 [app] [I] Started POST
"/my_plugin/api/v2/discover_host" for 10.129.206.169 at 2017-03-13 11:30:56
+0100

11:30:56 rails.1 | 2017-03-13T11:30:56 522230d8 [app] [I] Processing by
Api::V2::DiscoveryController#discover_host as /

11:30:56 rails.1 | 2017-03-13T11:30:56 522230d8 [app] [I] Parameters:
{"host"=>"[FILTERED]", "location_id"=>1, "organization_id"=>1,
"password"=>"[FILTERED]", "apiv"=>"v2", "discovery"=>{"host"=>"[FILTERED]",
"location_id"=>1, "organization_id"=>1, "password"=>"[FILTERED]"}}

11:30:56 rails.1 | 2017-03-13T11:30:56 522230d8 [app] [D] Setting current
user thread-local variable to admin

11:30:56 rails.1 | 2017-03-13T11:30:56 522230d8 [foreman-tasks/dynflow]
[D] ExecutionPlan 33714ed3-a05e-4eb6-b5d8-88b559e44d8c pending >>
planning

11:30:57 rails.1 | 2017-03-13T11:30:56 522230d8 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 1 pending >>
running in phase Plan Actions::ForemanAnsible::PlayHostRole

11:30:58 rails.1 | 2017-03-13T11:30:58 522230d8 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 2 pending >>
running in phase Plan ForemanAnsibleCore::Actions::RunPlaybook

11:30:58 rails.1 | 2017-03-13T11:30:58 522230d8 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 2 running >>
success in phase Plan ForemanAnsibleCore::Actions::RunPlaybook

11:30:58 rails.1 | 2017-03-13T11:30:58 522230d8 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 1 running >>
success in phase Plan Actions::ForemanAnsible::PlayHostRole

11:30:58 rails.1 | 2017-03-13T11:30:58 522230d8 [foreman-tasks/dynflow]
[D] ExecutionPlan 33714ed3-a05e-4eb6-b5d8-88b559e44d8c planning >>
planned

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] ExecutionPlan 33714ed3-a05e-4eb6-b5d8-88b559e44d8c planned >>
running

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 pending >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 pending >>
running in phase Run ForemanAnsibleCore::Actions::RunPlaybook

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 running >>
suspended in phase Run ForemanAnsibleCore::Actions::RunPlaybook

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] start runner fc611544-441c-412b-b396-58765309bca8

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] [foreman_ansible] - Initializing Ansible Runner

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] [foreman_ansible] - Running command
[{"JSON_INVENTORY_FILE"=>"/tmp/d20170313-8939-1ecph8h/foreman-inventories/fc611544-441c-412b-b396-58765309bca8"},
"ansible-playbook", "-i", "/root/foreman_ansible/bin/json_inventory.sh",
"/tmp/d20170313-8939-1ecph8h/foreman-playbook-fc611544-441c-412b-b396-58765309bca8.yml"]

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:30:58 rails.1 | 2017-03-13T11:30:58 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:00 rails.1 | 2017-03-13T11:31:00 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:00 rails.1 | 2017-03-13T11:31:00 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:00 rails.1 | 2017-03-13T11:31:00 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:00 rails.1 | 2017-03-13T11:31:00 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 got event
#<ForemanTasksCore::runner::Update:0x007f02a0e30518>

11:31:00 rails.1 | 2017-03-13T11:31:00 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 suspended >>
running in phase Run ForemanAnsibleCore::Actions::RunPlaybook

11:31:00 rails.1 | 2017-03-13T11:31:00 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 running >>
suspended in phase Run ForemanAnsibleCore::Actions::RunPlaybook

11:31:01 rails.1 | 2017-03-13T11:31:01 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 got event
Dynflow::Action::Polling::Poll

11:31:01 rails.1 | 2017-03-13T11:31:01 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:01 rails.1 | 2017-03-13T11:31:01 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:02 rails.1 | 2017-03-13T11:31:02 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:02 rails.1 | 2017-03-13T11:31:02 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:02 rails.1 | 2017-03-13T11:31:02 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:04 rails.1 | 2017-03-13T11:31:04 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:04 rails.1 | 2017-03-13T11:31:04 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:04 rails.1 | 2017-03-13T11:31:04 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 got event
Dynflow::Action::Polling::Poll

11:31:04 rails.1 | 2017-03-13T11:31:04 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:04 rails.1 | 2017-03-13T11:31:04 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:04 rails.1 | 2017-03-13T11:31:04 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:06 rails.1 | 2017-03-13T11:31:06 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:06 rails.1 | 2017-03-13T11:31:06 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:06 rails.1 | 2017-03-13T11:31:06 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 got event
Dynflow::Action::Polling::Poll

11:31:06 rails.1 | 2017-03-13T11:31:06 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:06 rails.1 | 2017-03-13T11:31:06 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:06 rails.1 | 2017-03-13T11:31:06 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:08 rails.1 | 2017-03-13T11:31:08 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:08 rails.1 | 2017-03-13T11:31:08 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:08 rails.1 | 2017-03-13T11:31:08 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:08 rails.1 | 2017-03-13T11:31:08 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 got event
#<ForemanTasksCore::runner::Update:0x007f02a136e390>

11:31:08 rails.1 | 2017-03-13T11:31:08 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 suspended >>
running in phase Run ForemanAnsibleCore::Actions::RunPlaybook

11:31:08 rails.1 | 2017-03-13T11:31:08 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 3 running >>
suspended in phase Run ForemanAnsibleCore::Actions::RunPlaybook

11:31:09 rails.1 | 2017-03-13T11:31:09 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 got event
Dynflow::Action::Polling::Poll

11:31:09 rails.1 | 2017-03-13T11:31:09 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:09 rails.1 | 2017-03-13T11:31:09 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:10 rails.1 | 2017-03-13T11:31:10 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:10 rails.1 | 2017-03-13T11:31:10 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:10 rails.1 | 2017-03-13T11:31:10 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:11 rails.1 | 2017-03-13T11:31:11 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 got event
Dynflow::Action::Polling::Poll

11:31:11 rails.1 | 2017-03-13T11:31:11 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:11 rails.1 | 2017-03-13T11:31:11 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:12 rails.1 | 2017-03-13T11:31:12 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:12 rails.1 | 2017-03-13T11:31:12 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:12 rails.1 | 2017-03-13T11:31:12 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:14 rails.1 | 2017-03-13T11:31:14 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:14 rails.1 | 2017-03-13T11:31:14 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:14 rails.1 | 2017-03-13T11:31:14 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 got event
Dynflow::Action::Polling::Poll

11:31:14 rails.1 | 2017-03-13T11:31:14 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:14 rails.1 | 2017-03-13T11:31:14 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:14 rails.1 | 2017-03-13T11:31:14 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:16 rails.1 | 2017-03-13T11:31:16 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:16 rails.1 | 2017-03-13T11:31:16 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:16 rails.1 | 2017-03-13T11:31:16 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 got event
Dynflow::Action::Polling::Poll

11:31:16 rails.1 | 2017-03-13T11:31:16 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:16 rails.1 | 2017-03-13T11:31:16 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:16 rails.1 | 2017-03-13T11:31:16 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:18 rails.1 | 2017-03-13T11:31:18 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:18 rails.1 | 2017-03-13T11:31:18 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:18 rails.1 | 2017-03-13T11:31:18 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:20 rails.1 | 2017-03-13T11:31:20 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:20 rails.1 | 2017-03-13T11:31:20 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:20 rails.1 | 2017-03-13T11:31:20 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 got event
Dynflow::Action::Polling::Poll

11:31:20 rails.1 | 2017-03-13T11:31:20 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:20 rails.1 | 2017-03-13T11:31:20 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:20 rails.1 | 2017-03-13T11:31:20 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:22 rails.1 | 2017-03-13T11:31:22 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:22 rails.1 | 2017-03-13T11:31:22 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:22 rails.1 | 2017-03-13T11:31:22 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 got event
Dynflow::Action::Polling::Poll

11:31:22 rails.1 | 2017-03-13T11:31:22 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:22 rails.1 | 2017-03-13T11:31:22 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:22 rails.1 | 2017-03-13T11:31:22 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:24 rails.1 | 2017-03-13T11:31:24 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:24 rails.1 | 2017-03-13T11:31:24 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:24 rails.1 | 2017-03-13T11:31:24 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:26 rails.1 | 2017-03-13T11:31:26 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:26 rails.1 | 2017-03-13T11:31:26 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:26 rails.1 | 2017-03-13T11:31:26 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 got event
Dynflow::Action::Polling::Poll

11:31:26 rails.1 | 2017-03-13T11:31:26 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:26 rails.1 | 2017-03-13T11:31:26 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:26 rails.1 | 2017-03-13T11:31:26 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:28 rails.1 | 2017-03-13T11:31:28 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:28 rails.1 | 2017-03-13T11:31:28 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:28 rails.1 | 2017-03-13T11:31:28 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 got event
Dynflow::Action::Polling::Poll

11:31:28 rails.1 | 2017-03-13T11:31:28 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:28 rails.1 | 2017-03-13T11:31:28 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:28 rails.1 | 2017-03-13T11:31:28 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:30 rails.1 | 2017-03-13T11:31:30 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:30 rails.1 | 2017-03-13T11:31:30 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:30 rails.1 | 2017-03-13T11:31:30 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:32 rails.1 | 2017-03-13T11:31:32 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:32 rails.1 | 2017-03-13T11:31:32 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:32 rails.1 | 2017-03-13T11:31:32 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 got event
Dynflow::Action::Polling::Poll

11:31:32 rails.1 | 2017-03-13T11:31:32 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:32 rails.1 | 2017-03-13T11:31:32 ffd3b366 [foreman-tasks/dynflow]
[D] Step 33714ed3-a05e-4eb6-b5d8-88b559e44d8c: 5 running >>
suspended in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:32 rails.1 | 2017-03-13T11:31:32 ffd3b366 [foreman-tasks/dynflow]
[D] planning to refresh fc611544-441c-412b-b396-58765309bca8

11:31:34 rails.1 | 2017-03-13T11:31:34 ffd3b366 [foreman-tasks/dynflow]
[D] refresh runner fc611544-441c-412b-b396-58765309bca8

11:31:34 rails.1 | 2017-03-13T11:31:34 ffd3b366 [foreman-tasks/dynflow]
[D] refreshing runner

11:31:34 rails.1 | 2017-03-13T11:31:34 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 got event
Dynflow::Action::Polling::Poll

11:31:34 rails.1 | 2017-03-13T11:31:34 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 suspended >>
running in phase Run Actions::ForemanAnsible::PlayHostRole

11:31:34 rails.1 | 2017-03-13T11:31:34 ffd3b366 [foreman-tasks/dynflow]
[D] Step 5437635c-7606-4c18-a103-8ad9ae2d5945: 5 running >>
suspended in phase Run