Remote Action Run Ansible roles stuck pending > 24h

Support
1

Problem:

Scheduled ansible runs stuck in pending state for over 24 hours

Expected outcome:

Scheduled ansible runs do not get stuck

Foreman and Proxy versions:

foreman-3.3.0-1.el7.noarch
foreman-proxy-3.3.0-1.el7.noarch

Foreman and Proxy plugin versions:

Distribution and version:

CentOS Linux release 7.9.2009 (Core)

Other relevant data:

For now we have only scheduled ansible runs on 20 hosts. I have no clue why they get stuck once a day and I also have no starting point where to look for logs. It seems that the jobs always get stuck during 05:20 in the morning. All ansible runs before and after are running fine. In the Jobs overview the job is listed as running 100%.

2

@aruzicka FYI, I think we’ve seen this last week

3

What do you see in output of the stuck job?

Also please do post logs. /var/log/foreman/production.log from Foreman, /var/log/foreman-proxy/proxy.log from the proxy through which the jobs go. If you could bump log level to debug, reproduce it again and then post the logs that would be even better.

4

Hi,

thanks for your reply. I will do that. Usually the issue occurs early in the morning. I will bump the log level to debug and then post the logs.

Cheers,
Oliver

5

Good morning,

Yesterday I changed the log level to debug on the proxy and the foreman server and restarted the services:

foreman-maintain service restart

After the restart the stuck jobs came back. I force cancelled them before. So now I have 61 running / pending ansbile run jobs. What is interesting, is that the jobs are stuck at 5:20 and 5:40 AM. So there is a pattern. :slight_smile:

Best Regards,
Oliver

6

So I dug a little deeper. There were no new stuck jobs. The last ones date back to 3 days ago (09/11).

2022-09-14 09_07_12-Tasks
2022-09-14 09_07_12-Tasks1656×818 118 KB

Back then there was no debug logging enabled, please let me know if you want me to upload the full logs anyway. Here are some snippets from both logs.The timezone on the foreman and proxy is UTC so if the job is failed at around 05:20 that is 03:20 on the host. I do see some errors around that time. Don’t know if they are related:

foreman:

2022-09-11T03:25:09 [E|app|bc1ba360] exception when talking to a remote client: Katello::Resources::Candlepin::Consumer: 404 Not Found {"displayMessage":"Consumer with id 8870b0b1-7723-4b96-99bd-a0b6bd1c7792 could not be found.","requestUuid":"ca43e5e5-b42a-402c-9420-0f413c0eb229"} (GET /candlepin/consumers/8870b0b1-7723-4b96-99bd-a0b6bd1c7792) RestClient::NotFound: Katello::Resources::Candlepin::Consumer: 404 Not Found {"displayMessage":"Consumer with id 8870b0b1-7723-4b96-99bd-a0b6bd1c7792 could not be found.","requestUuid":"ca43e5e5-b42a-402c-9420-0f413c0eb229"} (GET /candlepin/consumers/8870b0b1-7723-4b96-99bd-a0b6bd1c7792)
 bc1ba360 | Body: {"displayMessage":"Consumer with id 8870b0b1-7723-4b96-99bd-a0b6bd1c7792 could not be found.","requestUuid":"ca43e5e5-b42a-402c-9420-0f413c0eb229"}
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
 bc1ba360 | /opt/rh/rh-ruby27/root/usr/share/ruby/net/http.rb:933:in `start'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/katello/http_resource.rb:92:in `issue_request'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/katello/http_resource.rb:47:in `block (2 levels) in singleton class'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/katello/resources/candlepin/consumer.rb:18:in `get'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/controllers/katello/api/rhsm/candlepin_dynflow_proxy_controller.rb:56:in `block in find_host'
 bc1ba360 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
 bc1ba360 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/controllers/katello/api/rhsm/candlepin_dynflow_proxy_controller.rb:56:in `find_host'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:428:in `block in make_lambda'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:181:in `block (2 levels) in halting_and_conditional'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>
bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:182:in `block in halting_and_conditional'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in `block in invoke_before'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in `each'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in `invoke_before'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:110:in `block in run_callbacks'
 bc1ba360 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 bc1ba360 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 bc1ba360 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications.rb:180:in `block in instrument'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications.rb:180:in `instrument'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/controllers/katello/concerns/api/api_controller.rb:50:in `process_action'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/base.rb:136:in `process'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/rendering.rb:39:in `process'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal.rb:190:in `dispatch'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal.rb:254:in `dispatch'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:32:in `each'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/engine.rb:527:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/railtie.rb:190:in `public_send'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/railtie.rb:190:in `method_missing'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:32:in `each'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/lib/katello/middleware/event_daemon.rb:10:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 bc1ba360 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in `call'

foreman-proxy:

2022-09-11T03:23:04  [E] error while dispatching request to runner 8d7b5139-de79-4df5-af14-512e81dd31fd:Errno::ECHILD No child processes:
 /opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.8.1/lib/smart_proxy_dynflow/process_manager.rb:162:in `wait2'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.8.1/lib/smart_proxy_dynflow/process_manager.rb:162:in `finish'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.8.1/lib/smart_proxy_dynflow/process_manager.rb:124:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.8.1/lib/smart_proxy_dynflow/runner/process_manager_command.rb:35:in `refresh'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.8.1/lib/smart_proxy_dynflow/runner/base.rb:21:in `run_refresh'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.8.1/lib/smart_proxy_dynflow/runner/dispatcher.rb:40:in `refresh_runner'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/actor.rb:13:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.8.1/lib/smart_proxy_dynflow/runner/dispatcher.rb:24:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/actor.rb:122:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/actor.rb:56:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:353:in `run_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:342:in `block (3 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `block (2 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-09-11T03:23:04 5bf8be9a [E] Script execution failed

Best Regards,
Oliver

7

Huh, that’s interesting. The one from production.log is probably unrelated, the other issue is probably why the jobs are getting stuck. It should be fairly straightforward to fix, although I can’t really explain why it happens.

8

Thanks for the quick reply. I was about to upgrade to foreman 3.4 and Katello 4.6 hoping that my issue is already fixed in the newer releases, but I just noticed that EL7 as host OS is no longer supported. And it seems that there are no instructions for an upgrade path because the docs refer to upgrading to foreman 3.4 / Katello 4.6 which is actually no longer supported under EL7. I will raise a new questions for this. Until then I just wait for further instructions from your side :slight_smile:

9

Ok, I guess I have to use the migration path:

https://docs.theforeman.org/3.4/Upgrading_and_Updating/index-katello.html#migrating-project-to-a-new-el-system_upgrade-guide

10

Since you’re able to reproduce the issue reliably, would you be willing to test a patch[1]?

[1] - https://github.com/theforeman/smart_proxy_dynflow/pull/113

11

Sure. Do I just need to replace the two files on the proxy or also on the foreman host?

Cheers

12

It needs to happen on the proxy that is used for ansible, but note that it might be co-located on the same machine as foreman. If you can, it would be better if you could replace it everywhere, just to be sure.

13

Ok, I will replace them on the affected proxy for now. How to deal with the stuck jobs. I force cancelled them and they re-appeared after restarting the services. :slight_smile: is there a way to get rid of them completely?

14

You should be able to abort them from job details, that should make them stop for good.

15

Aborting from the UI didn’t work somehow. After restarting the service the tasks reappeared. I have now killed them via foreman rake-console.

ForemanTasks::Task.where(:state => :running).where(:result => :pending).destroy_all

They have not reappeared yet. I will keep an eye if the problem pops up again, if not your fix is good. :slight_smile:

16

I would advise against going to rails console (or directly to the database) and removing tasks that way. Chances are you’ll leave orphaned records behind. If you are absolutely sure you have to remove tasks, then please at least use the foreman_tasks:cleanup rake task, which should take care of the orphans.

At this point, please do run the following to remove the orphans you might have.

foreman-rake foreman_tasks:cleanup TASK_SEARCH=state\ \!=\ stopped STATES=stopped VERBOSE=1 NOOP=1

Running it like this will only print what it would do, chances are it would remove some orhpans, but shouldn’t do anything else. If you’re satisfied with what it would do, drop the NOOP=1 suffix and run it again.

17

Thanks for pointing this out. I ran it and it removed 333 taks.

18

Did it help?

19

Hi,

yes I guess it helped. I added a few more scheduled ansible runs and also one on a proxy where I have not implemented the patch yet. On this one the jobs were stuck and pending. :slight_smile:

Thanks a lot for your quick fix.

Cheers,
Oliver