Task in state 'pending' and result 'success' cannot be destroyed

poulterer · November 4, 2020, 7:04pm

Problem:
A task lock is blocking several actions and the associated task cannot be destroyed.

Expected outcome:
Destroy the task and clear the lock.

Foreman and Proxy versions:
foreman 2.0.3
foreman-proxy 2.0.3
katello 3.15.3
pulp-server 2.21.4

Distribution and version:
CentOS 7 current

Other relevant data:

production.log

Please inspect their state, fix their errors and resume them.

Required lock: read
Conflicts with tasks:
- https://s2-foreman.s2.internal//foreman_tasks/tasks/10ed47e0-03fe-4e6a-b641-0e5cfa064774 (ForemanTasks::Lock::LockConflict)
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/models/foreman_tasks/lock.rb:44:in `block in <class:Lock>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:426:in `instance_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:426:in `block in make_lambda'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:606:in `block (2 levels) in default_terminator'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:605:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:605:in `block in default_terminator'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:199:in `block in halting'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:513:in `block in invoke_before'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:513:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:513:in `invoke_before'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:131:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:816:in `_run_validate_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations.rb:409:in `run_validations!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations/callbacks.rb:118:in `block in run_validations!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:132:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:816:in `_run_validation_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations/callbacks.rb:118:in `run_validations!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations.rb:339:in `valid?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/validations.rb:67:in `valid?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/validations.rb:84:in `perform_validations'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/validations.rb:52:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:315:in `block in save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:252:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:315:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/suppressor.rb:48:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/models/foreman_tasks/lock.rb:90:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/models/foreman_tasks/lock.rb:90:in `lock!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/helpers/lock.rb:13:in `lock!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/entry_action.rb:42:in `action_subject'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/actions/katello/content_view/remove.rb:21:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:517:in `block (3 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:31:in `with_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:516:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:420:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:515:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:514:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:285:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:6:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:284:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:211:in `block (2 levels) in plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/coordinator.rb:326:in `acquire'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:209:in `block in plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:208:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:208:in `plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:204:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:180:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:23:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:29:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:27:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:54:in `async_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks/triggers.rb:18:in `async_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/controllers/katello/api/v2/content_views_controller.rb:201:in `remove'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/middleware/event_daemon.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
[ concurrent-ruby ]
2020-11-04T10:06:31 [I|bac|] Task {label: Actions::Katello::ContentView::Remove, id: bb2ba877-af7a-4f8c-b79e-59e5227d3258, execution_plan_id: 5b9b42d9-ef5c-4659-a8c5-b595d7786235} state changed: stopped  result: error
2020-11-04T10:06:31 [I|bac|] Task {label: Actions::Katello::ContentView::Remove, id: bb2ba877-af7a-4f8c-b79e-59e5227d3258, execution_plan_id: 5b9b42d9-ef5c-4659-a8c5-b595d7786235} state changed: stopped  result: error
2020-11-04T10:06:31 [E|app|] ForemanTasks::Lock::LockConflict: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: read
Conflicts with tasks:
- https://s2-foreman.s2.internal//foreman_tasks/tasks/10ed47e0-03fe-4e6a-b641-0e5cfa064774
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/models/foreman_tasks/lock.rb:44:in `block in <class:Lock>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:426:in `instance_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:426:in `block in make_lambda'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:606:in `block (2 levels) in default_terminator'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:605:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:605:in `block in default_terminator'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:199:in `block in halting'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:513:in `block in invoke_before'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:513:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:513:in `invoke_before'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:131:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:816:in `_run_validate_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations.rb:409:in `run_validations!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations/callbacks.rb:118:in `block in run_validations!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:132:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:816:in `_run_validation_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations/callbacks.rb:118:in `run_validations!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/validations.rb:339:in `valid?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/validations.rb:67:in `valid?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/validations.rb:84:in `perform_validations'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/validations.rb:52:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:315:in `block in save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:252:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:315:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/suppressor.rb:48:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/models/foreman_tasks/lock.rb:90:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/models/foreman_tasks/lock.rb:90:in `lock!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/helpers/lock.rb:13:in `lock!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/entry_action.rb:42:in `action_subject'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/actions/katello/content_view/remove.rb:21:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:517:in `block (3 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:31:in `with_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:516:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:420:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:515:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:514:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:285:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:6:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:284:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:211:in `block (2 levels) in plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/coordinator.rb:326:in `acquire'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:209:in `block in plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:208:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:208:in `plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:204:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:180:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:23:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:29:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:27:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:54:in `async_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks/triggers.rb:18:in `async_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/controllers/katello/api/v2/content_views_controller.rb:201:in `remove'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/middleware/event_daemon.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

poulterer · November 4, 2020, 9:02pm

We were able to clear this in Postgres using:
su -c "psql -d foreman -c \"DELETE FROM foreman-tasks_tasks WHERE id in(SELECT id FROM foreman_tasks_tasks WHERE state = 'pending' AND result = 'success');\"" postgres
We then had to clear several other tasks that were stalled but all is now well.

aruzicka · November 5, 2020, 9:24am

I would discourage from doing so. If you just go into the db and start deleting things you deem fit, you’ll end up with orphaned records, which could cause issues in the long run.

If you’re 100% sure you really really need to destroy a task, there’s foreman-rake foreman_tasks:cleanup which can be used to destroy tasks and deals with the orphaned objects as well.

poulterer · November 5, 2020, 2:47pm

@aruzicka - Thanks for your response.
We had, of course, already attempted
foreman-rake foreman_tasks:cleanup
and
foreman-rake console <<< $(echo "'ForemanTasks::Task.find("10ed47e0-03fe-4e6a-b641-0e5cfa064774").destroy'")
to no effect. We restarted foreman by
foreman-maintain service restart
and rebooted the system, trying all of the above again, to no effect. We are aware of the inherent dangers of direct database manipulation but needed to resolve this issue quickly as we have an upcoming patch cycle. Prior to performing this action, we took a system snapshot and db:dump, so reversion was possible if it went sideways.
We will update to 2.1 after our current patch cycle.
I would discourage you from dispensing cavalier, condescending advice without having all the facts. If you just do so as you deem fit, you’ll end up with irritated users, which could cause issues in the long run.

aruzicka · November 5, 2020, 5:02pm

I’m sorry to hear you’re having issues and I get how frustrating it can be.

All I wanted to state is there is a tool which should be a go-to solution for this kind of situations. If there’s an issue with the tool, we’d like to hear about it so we can fix it so it can remain the go-to tool without everyone having to invent their own solution.

What arguments did you supply to foreman-rake foreman_tasks:cleanup when you attempted to use it and what was its output?

What was the output when you tried running foreman-rake console <<< $(echo "'ForemanTasks::Task.find("10ed47e0-03fe-4e6a-b641-0e5cfa064774").destroy'")?

poulterer · November 5, 2020, 6:21pm

No arguments were supplied to
foreman-rake foreman_tasks:cleanup
and the output from
foreman-rake console <<< $(echo "'ForemanTasks::Task.find("10ed47e0-03fe-4e6a-b641-0e5cfa064774").destroy'")
was two lines indicating the task had been destroyed. However,
hammer task list --search id=10ed47e0-03fe-4e6a-b641-0e5cfa064774
showed this task still present. Since nothing seemed able to touch the task, we opted for the previously described procedure.

aruzicka · November 5, 2020, 7:54pm

When you run foreman-rake foreman_tasks:cleanup without any arguments, it performs a rule-based cleanup based on rules defined somewhere in /etc/foreman/foreman-tasks.yml (the path might be slightly different, I’m a bit foggy on the details and don’t have an environment where I could check right now). Unless you have your own custom rules set up, the default ones definitely wouldn’t target the task you wanted to be rid of.

To make it not use the rules, you can provide arguments to control which tasks would be matched. I won’t go over all of them, but to reuse your example, the following snippet should do the trick.

foreman-rake foreman_tasks:cleanup TASK_SEARCH="id = 10ed47e0-03fe-4e6a-b641-0e5cfa064774" STATES=all

poulterer · November 5, 2020, 8:50pm

I appreciate the information regarding foreman_tasks:cleanup. But, how does that explain that we were unable to destroy the task when explicitly named?

aruzicka · November 5, 2020, 9:04pm

Well, if the command you stated is really what you run without any changes, then it was evaluated to 'ForemanTasks::Task.find("10ed47e0-03fe-4e6a-b641-0e5cfa064774").destroy' as a string, not the actual code it was meant to represent. If not, then I’m afraid I’m out of ideas.

poulterer · November 5, 2020, 9:06pm

It has always worked previously to destroy a specific task.