Failing tasks. "RuntimeError: No pulpcore workers are running"

Support
,
1

For the past few weeks “Pulp disk space notification” and repository sync plans started sporadically failing. In both cases the error is stated as ‘no pupcore workers are running’

The system had been installed and running for about a year - no problems whatsoever. No updates or any alternations (appart from adding sometimes content views, registering new hosts) of the system had been done recently. The issues just started appearing suddenly.

Any tips for troubleshooting appreciated

Foreman and Proxy versions: 3.1.2

Katello version: 4.3.1-1

Distribution and version: Rocky Linux 8.5

Other relevant data:
Backtrace from one failed sync plan


/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/backend_services_check.rb:17:in `block in plan'
/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/backend_services_check.rb:15:in `each'
/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/backend_services_check.rb:15:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/load_setting_values.rb:15:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:34:in `with_current_request_id'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:41:in `with_current_user'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:524:in `block (2 levels) in execute_plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:387:in `switch_flow'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:428:in `concurrence'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:523:in `block in execute_plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:522:in `execute_plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:456:in `plan_action'
/usr/share/gems/gems/katello-4.3.1/app/lib/actions/katello/repository/sync.rb:43:in `block in plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:387:in `switch_flow'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:433:in `sequence'
/usr/share/gems/gems/katello-4.3.1/app/lib/actions/katello/repository/sync.rb:39:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:525:in `block (3 levels) in execute_plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/load_setting_values.rb:15:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:34:in `with_current_request_id'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:36:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:41:in `with_current_user'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:524:in `block (2 levels) in execute_plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:387:in `switch_flow'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:428:in `concurrence'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:523:in `block in execute_plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:522:in `execute_plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:289:in `block (2 levels) in plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:387:in `switch_flow'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:377:in `with_planning_scope'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:288:in `block in plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/common/transaction.rb:6:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:287:in `plan'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:219:in `block (2 levels) in plan_with_options'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator.rb:326:in `acquire'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:217:in `block in plan_with_options'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:216:in `tap'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:216:in `plan_with_options'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:84:in `block in trigger'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:178:in `trigger'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:84:in `trigger'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/bulk_action.rb:54:in `block in create_sub_plans'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/bulk_action.rb:53:in `map'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/bulk_action.rb:53:in `create_sub_plans'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:41:in `spawn_plans'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_bulk_sub_plans.rb:71:in `spawn_plans'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:37:in `initiate'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_bulk_sub_plans.rb:36:in `initiate'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:17:in `block in run'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_sub_plans.rb:12:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/with_bulk_sub_plans.rb:24:in `run'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/bulk_action.rb:21:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2

Do you see anything in the system logs about pulpcore workers being killed? Perhaps you’re on the upper limits of your system’s RAM ?

3

Hello and thank you for the reply!

The Foreman system runs on a KVM machine with dedicated 64 gb RAM and 12 logical cores. It’s serving about 150 contents hosts, with handful of content views and subscriptions defined. I’ve never seen more than 14-16 gigs of RAM being used totally on this machine. And it has been very fast and responsive until this suddenly started happening.

While I can’t find anything in the logs about pulpcore workers being killed I’m finding a lot of rows of this type in log journal for every single worker:

pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Worker workerid@hostname is back online.

As matter of fact they appear every 5 minutes or so. Does ‘back online’ imply the worker for some reason was killed?