Actions::Pulp3::Repository::RefreshDistribution (waiting for Pulp to start the task)

Problem:
Sync, New version, Upload,… always hangs at Actions::Pulp3::Repository::RefreshDistribution (waiting for Pulp to start the task)

Expected outcome:

Foreman and Proxy versions:
Foreman: 2.3.5
Foreman and Proxy plugin versions:
foreman-tasks: 3.0.6
foreman_remote_execution: 4.2.2
katello: 3.18.4

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:
Restarted foreman services via: foreman-maintain service restart
Cleaned foreman: foreman-rake foreman_tasks:cleanup
Rebooted server.
Still stuck at RefreshDistributin

Maybe the Pulp workers are failing to start. If so, systemctl status pulpcore-worker* might give you some more information.

One of the more likely candidates might be if your Pulp DB migrations aren’t fully applied. (This can sometimes happen after yum update and is therefore often overlooked).

1 Like

All pulpcore-workers are working.
The issue started with a failed sync.

In that case the full error backtrace of the failed sync might provide some clues.

Hi,

I have tried to upload an very small rpm.
This is the error:

I can not upload the log, so I pasted it here:


2021-08-26T09:14:42 [I|app|e9c18633] Started POST "/katello/api/v2/repositories/1/upload_content" for 10.8.1.4 at 2021-08-26 09:14:42 +0200
2021-08-26T09:14:42 [I|app|e9c18633] Processing by Katello::Api::V2::RepositoriesController#upload_content as HTML
2021-08-26T09:14:42 [I|app|e9c18633]   Parameters: {"content"=>[#<ActionDispatch::Http::UploadedFile:0x000000001ce0e248 @tempfile=#<Tempfile:/tmp/RackMultipart20210826-10844-116l4av.rpm>, @original_filename="katello-host-tools-3.5.4-1.el7.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"katello-host-tools-3.5.4-1.el7.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">], "authenticity_token"=>"waJ15z4Dq+G/BnbsQZLl+ibqGHACoE0We6DVjhG3QRxeKk2kjBcp9z9iD7kuX32ZEp+X9k9YUSOPxDJHZTdB8A==", "api_version"=>"v2", "id"=>"1"}
2021-08-26T09:14:42 [I|bac|e9c18633] Task {label: Actions::Katello::Repository::UploadFiles, id: e6663012-6075-4152-9f2c-9d2c355d0693, execution_plan_id: c0529ce0-b994-42fa-a95a-3b42dc62d3ba} state changed: planning 
2021-08-26T09:14:43 [I|bac|] Task {label: Actions::Katello::Repository::UploadFiles, id: e6663012-6075-4152-9f2c-9d2c355d0693, execution_plan_id: c0529ce0-b994-42fa-a95a-3b42dc62d3ba} state changed: planned 
2021-08-26T09:14:43 [I|bac|] Task {label: Actions::Katello::Repository::UploadFiles, id: e6663012-6075-4152-9f2c-9d2c355d0693, execution_plan_id: c0529ce0-b994-42fa-a95a-3b42dc62d3ba} state changed: running 
2021-08-26T09:14:50 [I|app|2f4bcf8e] Started GET "/notification_recipients" for 10.8.1.4 at 2021-08-26 09:14:50 +0200
2021-08-26T09:14:50 [I|app|2f4bcf8e] Processing by NotificationRecipientsController#index as JSON
2021-08-26T09:14:50 [I|app|2f4bcf8e] Completed 200 OK in 6ms (Views: 0.1ms | ActiveRecord: 0.9ms | Allocations: 2526)
2021-08-26T09:14:50 [I|aud|] Katello::Repository (1) update event on publication_href /pulp/api/v3/publications/rpm/rpm/0ed6307d-c1d1-438b-9db7-f9634d8c99eb/, /pulp/api/v3/publications/rpm/rpm/d0b6cf49-037d-46e9-9f65-f7e70a03e7e3/
...
2021-08-26T09:16:43 [E|app|] Timeout::Error: The time waiting for task e6663012-6075-4152-9f2c-9d2c355d0693 to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/lib/foreman_tasks.rb:39:in `block (2 levels) in trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/lib/foreman_tasks.rb:29:in `block in trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/lib/foreman_tasks.rb:27:in `trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/lib/foreman_tasks.rb:58:in `sync_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/lib/foreman_tasks/triggers.rb:22:in `sync_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/controllers/katello/api/v2/repositories_controller.rb:388:in `upload_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:195:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:139:in `run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `block in instrument'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `instrument'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/controllers/katello/concerns/api/api_controller.rb:40:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:136:in `process'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:39:in `process'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:190:in `dispatch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:254:in `dispatch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `serve'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `method_missing'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `serve'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/middleware/event_daemon.rb:10:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:101:in `run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:37:in `call_app'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:28:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/prevent_json_parsing.rb:12:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `method_missing'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/configuration.rb:228:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:713:in `handle_request'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:472:in `process_client'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:328:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-08-26T09:16:43 [I|app|] Completed 500 Internal Server Error in 120430ms (Views: 0.2ms | ActiveRecord: 37.3ms | Allocations: 207849)

Kind regards,

You are getting: Timeout::Error: The time waiting for task e6663012-6075-4152-9f2c-9d2c355d0693 to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)

This task is probably timing out since it is waiting for a Pulp task. (Tough a foreman_tasks_sync_task_timeout of 120s does not seem like very much, maybe try increasing this value?) So what we really need to find out, is what is going wrong with the Pulp task. It is possible to find out the ID of the underlying Pulp task from the UI of the failed Foreman task, and then retrieve the actual Pulp task using that ID.

PS: I am afraid I don’t know the exact commands for retrieving Pulp tasks. Since this is something that comes up often, I have opened a seperate thread for it here: How-To retrieve underlying Pulp tasks for Katello users Hopefully someone will share some step-by-step instructions soon…

Taking it from [Content-Migration] katello:correct_repositories and correct_puppet_environments fails blocking migration:

The low level method using curl is like this:

# curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://foreman.example.com/pulp/api/v3/tasks/31fee624-f68b-4650-91fd-71fc7938266c/' | python -m json.tool

Together with the Pulp 3 REST API documentation

When checking for task output just make sure not to confuse the foreman task id with the pulp3 task id as both are uuids.

I needed to use the katello certs:

curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.example.com/pulp/api/v3/tasks/ec34a9d6-b8f2-4370-8162-3dd3a5e8e528/' | python -m json.tool

The response is:

{
    "child_tasks": [],
    "created_resources": [],
    "error": null,
    "finished_at": null,
    "name": "pulpcore.app.tasks.base.general_update",
    "parent_task": null,
    "progress_reports": [],
    "pulp_created": "2021-08-26T07:14:50.453040Z",
    "pulp_href": "/pulp/api/v3/tasks/ec34a9d6-b8f2-4370-8162-3dd3a5e8e528/",
    "reserved_resources_record": [
        "/api/v3/distributions/"
    ],
    "started_at": null,
    "state": "waiting",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/fe235b2f-2905-4233-8f97-8a38447f975f/"
}

@ehelms @Justin_Sherrill @x9c4 is this our beloved “RQ-task-system decided to stop launching tasks” again?

Did we ever find a way to tell it to re-start the queue again?

Hi @evgeni @ehelms @Justin_Sherrill @x9c4 ,

Any news on the “RQ-task-system decided to stop launching tasks”?
What can I do the get the Pulp started again?

Kind regards

The first question to check is, whether that worker assigned to the task is still alive.
If not, the task is doomed. The way to thaw the system is by cancelling all waiting tasks.

1 Like

Hi, @x9c4

I’ve already restarted foreman completely and canceled all tasks.
But even then I still get the the “waiting” status.

Kind regards

Sorry, i didn’t specify, that i meant the pulp tasks. You need to perform a PATCH call with state=canceled on every task as described here:

https://docs.pulpproject.org/pulpcore/restapi.html#operation/tasks_cancel

Hi @x9c4 ,

I’ve canceled all the tasks. Restarted foreman, but still the same.
Next, I’ve canceled and deleted the tasks (some couldn’t be deleted), restarted foreman, but still the same.

I did an rpm upload to a repo:

sudo curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman/pulp/api/v3/tasks/603409a7-347b-43a6-8dfc-8a50d132a6e8/' | jq
{
  "pulp_href": "/pulp/api/v3/tasks/603409a7-347b-43a6-8dfc-8a50d132a6e8/",
  "pulp_created": "2021-09-01T09:20:59.425050Z",
  "state": "waiting",
  "name": "pulpcore.app.tasks.base.general_update",
  "started_at": null,
  "finished_at": null,
  "error": null,
  "worker": "/pulp/api/v3/workers/fe235b2f-2905-4233-8f97-8a38447f975f/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [],
  "created_resources": [],
  "reserved_resources_record": [
    "/api/v3/distributions/"
  ]
}

In attachement the relative foreman log.

What could be the next steps?

Kind regards,
foreman.log (24.9 KB)

Have you made sure all pulp tasks are canceled? The command should be something like this for each task waiting on the dead worker:

curl --cert <...> --key <...> --data '{state: "canceled"} -X PATCH https://<...>/pulp/api/v3/tasks/<UUID>/

Yes, I canceled all the waiting and failed tasks:

sudo curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman/pulp/api/v3/tasks/?limit=1500' | jq '.results[] | select(.state == "waiting") | .pulp_href' 

There are 2 failed tasks I can not cancel or delete:

$ sudo curl -s -X PATCH --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key -H "Content-Type: application/json" "https://foreman./pulp/api/v3/tasks/710c1b46-c045-4c5e-8420-004eb066792a/" --data '{"state":"canceled"}' | jq
{
  "pulp_href": "/pulp/api/v3/tasks/710c1b46-c045-4c5e-8420-004eb066792a/",
  "pulp_created": "2021-08-20T08:38:31.285944Z",
  "state": "failed",
  "name": "pulpcore.app.tasks.base.general_update",
  "started_at": null,
  "finished_at": "2021-08-20T08:40:38.754088Z",
  "error": {
    "traceback": "",
    "description": "Error 113 connecting to localhost:6379. No route to host."
  },
  "worker": "/pulp/api/v3/workers/fe235b2f-2905-4233-8f97-8a38447f975f/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [],
  "created_resources": [],
  "reserved_resources_record": [
    "/api/v3/distributions/"
  ]
}

Delete:

$ sudo curl -s -X DELETE --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key "https://foreman/pulp/api/v3/tasks/710c1b46-c045-4c5e-8420-004eb066792a/"
<h1>Server Error (500)</h1>c

Journal log:

Sep 01 11:59:19 foreman pulpcore-api[37447]: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/tasks/710c1b46-c045-4c5e-8420-004eb066792a/
Sep 01 11:59:19 foreman pulpcore-api[37447]: Traceback (most recent call last):
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
Sep 01 11:59:19 foreman pulpcore-api[37447]: response = get_response(request)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
Sep 01 11:59:19 foreman pulpcore-api[37447]: response = self.process_exception_by_middleware(e, request)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
Sep 01 11:59:19 foreman pulpcore-api[37447]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 01 11:59:19 foreman pulpcore-api[37447]: return view_func(*args, **kwargs)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 114, in view
Sep 01 11:59:19 foreman pulpcore-api[37447]: return self.dispatch(request, *args, **kwargs)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 505, in dispatch
Sep 01 11:59:19 foreman pulpcore-api[37447]: response = self.handle_exception(exc)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 465, in handle_exception
Sep 01 11:59:19 foreman pulpcore-api[37447]: self.raise_uncaught_exception(exc)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
Sep 01 11:59:19 foreman pulpcore-api[37447]: raise exc
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 502, in dispatch
Sep 01 11:59:19 foreman pulpcore-api[37447]: response = handler(request, *args, **kwargs)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/pulpcore/app/viewsets/task.py", line 94, in destroy
Sep 01 11:59:19 foreman pulpcore-api[37447]: return super().destroy(request, pk)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/rest_framework/mixins.py", line 91, in destroy
Sep 01 11:59:19 foreman pulpcore-api[37447]: self.perform_destroy(instance)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/rest_framework/mixins.py", line 95, in perform_destroy
Sep 01 11:59:19 foreman pulpcore-api[37447]: instance.delete()
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 141, in delete
Sep 01 11:59:19 foreman pulpcore-api[37447]: super().delete(*args, **kwargs)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
Sep 01 11:59:19 foreman sudo[44650]: pam_unix(sudo:session): session closed for user root
Sep 01 11:59:19 foreman pulpcore-api[37447]: collector.collect([self], keep_parents=keep_parents)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
Sep 01 11:59:19 foreman pulpcore-api[37447]: field.remote_field.on_delete(self, field, sub_objs, self.using)
Sep 01 11:59:19 foreman pulpcore-api[37447]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
Sep 01 11:59:19 foreman pulpcore-api[37447]: sub_objs
Sep 01 11:59:19 foreman pulpcore-api[37447]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Task' because they are referenced through a protected foreign key: 'TaskReservedResource.task'", <QuerySet [<TaskReservedResource: pk=afbf08d5-39e2-4f97-
Sep 01 11:59:19 foreman pulpcore-api[37447]: - - [01/Sep/2021:09:59:19 +0000] "DELETE /pulp/api/v3/tasks/710c1b46-c045-4c5e-8420-004eb066792a/ HTTP/1.1" 500 27 "-" "curl/7.29.0"

Can you try this:

# pulpcore-manager shell -c "from pulpcore.app.models import Task; Task.objects.get(pk='710c1b46-c045-4c5e-8420-004eb066792a').reserved_resources.clear()"

I receive an error:

# pulpcore-manager shell -c "from pulpcore.app.models import Task; Task.objects.get(pk='710c1b46-c045-4c5e-8420-004eb066792a').reserved_resources.clear()"
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 254, in <module>
    CONTENT_ORIGIN
NameError: name 'CONTENT_ORIGIN' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bin/pulpcore-manager", line 11, in <module>
    load_entry_point('pulpcore==3.7.6', 'console_scripts', 'pulpcore-manager')()
  File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/usr/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/lib/python3.6/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/usr/lib/python3.6/site-packages/pulpcore/app/apps.py", line 81, in ready
    self.import_viewsets()
  File "/usr/lib/python3.6/site-packages/pulpcore/app/apps.py", line 110, in import_viewsets
    from pulpcore.app.viewsets import NamedModelViewSet
  File "/usr/lib/python3.6/site-packages/pulpcore/app/viewsets/__init__.py", line 1, in <module>
    from .base import (  # noqa
  File "/usr/lib/python3.6/site-packages/pulpcore/app/viewsets/base.py", line 17, in <module>
    from pulpcore.app import tasks
  File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/__init__.py", line 1, in <module>
    from pulpcore.app.tasks import base, repository, upload  # noqa
  File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/upload.py", line 6, in <module>
    from pulpcore.app.serializers import ArtifactSerializer
  File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/__init__.py", line 42, in <module>
    from .exporter import (  # noqa
  File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/exporter.py", line 8, in <module>
    from pulpcore.app import models, settings
  File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 258, in <module>
    "CONTENT_ORIGIN is a required setting but it was not configured. This may be caused "
django.core.exceptions.ImproperlyConfigured: CONTENT_ORIGIN is a required setting but it was not configured. This may be caused by invalid read permissions of the settings file. Note that CONTENT_ORIGIN is set by the installer automatically.

can you add these to your environment?

export DJANGO_SETTINGS_MODULE=pulpcore.app.settings
export PULP_SETTINGS=/etc/pulp/settings.py

(Not sure if the path is right for a Katello installation.)

All the tasks are canceled and deleted.

Tried an upload and still got the error:

In the syslog:

Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38004@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38004@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38004@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38004@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38005@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38005@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38005@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38005@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '37960@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 37960@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '37960@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 37960@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-2[71125]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-7[71124]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-1[71126]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-1[71126]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-6[71137]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-6[71137]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:44 foreman pulpcore-worker-5[71127]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:44 foreman pulpcore-worker-5[71127]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:45 foreman pulpcore-worker-3[71138]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:45 foreman pulpcore-worker-3[71138]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:45 foreman pulpcore-worker-8[71136]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:45 foreman pulpcore-worker-8[71136]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.
Sep 01 15:54:45 foreman pulpcore-worker-4[71139]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '38067@foreman.sofico.be' has gone missing, removing from list of workers
Sep 01 15:54:45 foreman pulpcore-worker-4[71139]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 38067@foreman.sofico.be is missing. Canceling the tasks in its queue.

foreman/production.log:

2021-09-01T15:54:57 [E|bac|] Task canceled (Katello::Errors::Pulp3Error)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/execute_if_contents_changed.rb:5:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]