Tasks: Listen on candlepin events: already active

The “Listen on candlepin events” task continually appears to error out with:
RuntimeError: Action Actions::Candlepin::ListenOnCandlepinEvents is already active

This task should be continually running (or at least periodically a successful run.)

foreman-1.23.1-1.el7.noarch
katello-3.13.3-1.el7.noarch

CentOS7, fully patched

I’m at a loss, as I really haven’t managed to wrap my tiny little brain around the task subsystem.

Action:
Actions::Candlepin::ListenOnCandlepinEvent

Input:
{}

Output:
{}

Exception:
RuntimeError: Action Actions::Candlepin::ListenOnCandlepinEvents is already active

Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/singleton.rb:15:in `rescue in singleton_lock!'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/singleton.rb:12:in `singleton_lock!'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/common/singleton.rb:6:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.13.3/app/lib/actions/middleware/keep_locale.rb:7:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_request_id.rb:31:in `with_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:491:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan.rb:383:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:395:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:490:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:450:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:450:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:450:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:489:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:277:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan.rb:285:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan.rb:383:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan.rb:373:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan.rb:284:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan.rb:283:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/world.rb:204:in `block in plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/world.rb:202:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/world.rb:202:in `plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/world.rb:198:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/world.rb:174:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.16.3/lib/foreman_tasks.rb:23:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.13.3/app/lib/actions/candlepin/listen_on_candlepin_events.rb:54:in `ensure_running'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.13.3/lib/katello/engine.rb:91:in `block (2 levels) in <class:Engine>'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/configuration.rb:74:in `block in run_post_executor_init_hooks'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/configuration.rb:74:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/configuration.rb:74:in `run_post_executor_init_hooks'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails.rb:48:in `block in initialize!'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails.rb:37:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails.rb:37:in `initialize!'
/usr/share/foreman/config/application.rb:338:in `init_dynflow'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/usr/share/foreman/config/application.rb:313:in `block in <class:Application>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:69:in `block in execute_hook'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:51:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/application/finisher.rb:75:in `block in <module:Finisher>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:32:in `instance_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:32:in `run'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `each'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `call'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
/opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:60:in `run_initializers'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/application.rb:361:in `initialize!'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/daemon.rb:46:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/daemon.rb:78:in `block (2 levels) in run_background'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/application.rb:265:in `block in start_proc'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/daemonize.rb:84:in `call_as_daemon'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/application.rb:269:in `start_proc'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/application.rb:295:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/controller.rb:56:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons.rb:193:in `block in run_proc'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/cmdline.rb:88:in `catch_exceptions'
/opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons.rb:192:in `run_proc'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/daemon.rb:72:in `block in run_background'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/daemon.rb:71:in `times'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/rails/daemon.rb:71:in `run_background'
/usr/sbin/dynflowd:71:in `block in <main>'
/usr/sbin/dynflowd:67:in `chdir'
/usr/sbin/dynflowd:67:in `<main>'

Digging back through the dynflow console, it looks like I’ve been having this issue without realizing it since early October.

What’s my course of action? This is not production, I can completely tear down and rebuild this environment, although I’d really rather not have to.

And… finally figured out a bit of a solution.

https://projects.theforeman.org/issues/22915

Unfortunately, the syntax as listed there now fails under the current versions, as the equality operator between uuid and character varying fails. Here’s a working script:

cat <<EOF | sudo foreman-rake console
persistence = ForemanTasks.dynflow.world.persistence
adapter = persistence.adapter

batch_size = 5
total = adapter.db.fetch("select count(dynflow_execution_plans.uuid) from dynflo
w_execution_plans left join foreman_tasks_tasks on (dynflow_execution_plans.uuid
::text = foreman_tasks_tasks.external_id) where foreman_tasks_tasks.id IS NULL")
.first[:count]
deleted = 0
puts "about to delete #{total} execution plans"
while (plans_without_tasks = adapter.db.fetch("select dynflow_execution_plans.uu
id from dynflow_execution_plans left join foreman_tasks_tasks on (dynflow_execut
ion_plans.uuid::text = foreman_tasks_tasks.external_id) where foreman_tasks_task
s.id IS NULL LIMIT #{batch_size}").all.map { |x| x[:uuid] }) && !plans_without_t
asks.empty?
  persistence.delete_execution_plans({ 'uuid' => plans_without_tasks }, batch_si
ze)
  deleted += plans_without_tasks.count
  puts "deleted #{deleted} out of #{total}"
end
EOF

The trick is to add a cast for the uuid column to a text type.