Publishing a new Content view version produces an error nearly at 90% complete

Problem: Publishing a new Content view version produces an error nearly at 90% complete with the following: Error scheduling refresh job. (Katello::Errors::CandlepinError)

/var/log/foreman/production.log

2020-11-20T11:04:18 [I|bac|] Task {label: Actions::BulkAction, id: 7b21ecf3-ad1d-4c13-a5f3-205e667db706, execution_plan_id: b9111fb3-ac35-449e-9a70-b3be9d8e27d9} state changed: stopped  result: warning
2020-11-20T11:04:18 [I|bac|] Task {label: Actions::BulkAction, id: 7b21ecf3-ad1d-4c13-a5f3-205e667db706, execution_plan_id: b9111fb3-ac35-449e-9a70-b3be9d8e27d9} state changed: stopped  result: warning
2020-11-20T11:07:31 [I|bac|] Task {label: Actions::BulkAction, id: 1bf397f3-9e4f-43ca-b18f-ad5ca6320507, execution_plan_id: 290681be-aa29-4f9e-93bc-7c5912ccbf70} state changed: running
2020-11-20T11:07:31 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: ebb9ad6c-b66b-4866-865f-e689603c156c, execution_plan_id: dbb35557-8bcc-43f1-aba9-05fc8eaafa32} state changed: planning
2020-11-20T11:07:31 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: ebb9ad6c-b66b-4866-865f-e689603c156c, execution_plan_id: dbb35557-8bcc-43f1-aba9-05fc8eaafa32} state changed: planned
2020-11-20T11:07:31 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: ebb9ad6c-b66b-4866-865f-e689603c156c, execution_plan_id: dbb35557-8bcc-43f1-aba9-05fc8eaafa32} state changed: running
2020-11-20T11:08:12 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 99c3d253-b105-4a84-bdf6-7e5ae291e0cb, execution_plan_id: 129f5f54-bdd1-4908-874c-8572ff3549ec} state changed: planning
2020-11-20T11:08:12 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 99c3d253-b105-4a84-bdf6-7e5ae291e0cb, execution_plan_id: 129f5f54-bdd1-4908-874c-8572ff3549ec} state changed: planned
2020-11-20T11:08:12 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 99c3d253-b105-4a84-bdf6-7e5ae291e0cb, execution_plan_id: 129f5f54-bdd1-4908-874c-8572ff3549ec} state changed: running
2020-11-20T11:08:12 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 99c3d253-b105-4a84-bdf6-7e5ae291e0cb, execution_plan_id: 129f5f54-bdd1-4908-874c-8572ff3549ec} state changed: stopped  result: success
2020-11-20T11:08:12 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 99c3d253-b105-4a84-bdf6-7e5ae291e0cb, execution_plan_id: 129f5f54-bdd1-4908-874c-8572ff3549ec} state changed: stopped  result: success
2020-11-20T11:08:12 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: ebb9ad6c-b66b-4866-865f-e689603c156c, execution_plan_id: dbb35557-8bcc-43f1-aba9-05fc8eaafa32} state changed: stopped  result: success
2020-11-20T11:08:13 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: ebb9ad6c-b66b-4866-865f-e689603c156c, execution_plan_id: dbb35557-8bcc-43f1-aba9-05fc8eaafa32} state changed: stopped  result: success
2020-11-20T11:08:13 [I|bac|] Task {label: Actions::BulkAction, id: 1bf397f3-9e4f-43ca-b18f-ad5ca6320507, execution_plan_id: 290681be-aa29-4f9e-93bc-7c5912ccbf70} state changed: stopped  result: success
2020-11-20T11:08:13 [I|bac|] Task {label: Actions::BulkAction, id: 1bf397f3-9e4f-43ca-b18f-ad5ca6320507, execution_plan_id: 290681be-aa29-4f9e-93bc-7c5912ccbf70} state changed: stopped  result: success
2020-11-20T11:08:35 [I|bac|] Task {label: Actions::BulkAction, id: 2777be5f-9694-47ec-a03b-b6a189594b74, execution_plan_id: 14559ca8-217d-4d91-93c8-c76554bbaa76} state changed: running
2020-11-20T11:08:35 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e6f9809-2481-401e-a529-2f40fb6344ee, execution_plan_id: 7ebf116d-049e-4b62-99e2-b738467fbb7a} state changed: planning
2020-11-20T11:08:35 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e6f9809-2481-401e-a529-2f40fb6344ee, execution_plan_id: 7ebf116d-049e-4b62-99e2-b738467fbb7a} state changed: planned
2020-11-20T11:08:35 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e6f9809-2481-401e-a529-2f40fb6344ee, execution_plan_id: 7ebf116d-049e-4b62-99e2-b738467fbb7a} state changed: running
2020-11-20T11:10:49 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 3ab9bad2-0888-429a-a648-c0839be8d5aa, execution_plan_id: 89964f50-d129-44a6-957b-b2be05a80ddf} state changed: planning
2020-11-20T11:10:49 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 3ab9bad2-0888-429a-a648-c0839be8d5aa, execution_plan_id: 89964f50-d129-44a6-957b-b2be05a80ddf} state changed: planned
2020-11-20T11:10:49 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 3ab9bad2-0888-429a-a648-c0839be8d5aa, execution_plan_id: 89964f50-d129-44a6-957b-b2be05a80ddf} state changed: running
2020-11-20T11:10:49 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 3ab9bad2-0888-429a-a648-c0839be8d5aa, execution_plan_id: 89964f50-d129-44a6-957b-b2be05a80ddf} state changed: stopped  result: success
2020-11-20T11:10:49 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 3ab9bad2-0888-429a-a648-c0839be8d5aa, execution_plan_id: 89964f50-d129-44a6-957b-b2be05a80ddf} state changed: stopped  result: success
2020-11-20T11:10:50 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e6f9809-2481-401e-a529-2f40fb6344ee, execution_plan_id: 7ebf116d-049e-4b62-99e2-b738467fbb7a} state changed: stopped  result: success
2020-11-20T11:10:50 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e6f9809-2481-401e-a529-2f40fb6344ee, execution_plan_id: 7ebf116d-049e-4b62-99e2-b738467fbb7a} state changed: stopped  result: success
2020-11-20T11:10:50 [I|bac|] Task {label: Actions::BulkAction, id: 2777be5f-9694-47ec-a03b-b6a189594b74, execution_plan_id: 14559ca8-217d-4d91-93c8-c76554bbaa76} state changed: stopped  result: success
2020-11-20T11:10:50 [I|bac|] Task {label: Actions::BulkAction, id: 2777be5f-9694-47ec-a03b-b6a189594b74, execution_plan_id: 14559ca8-217d-4d91-93c8-c76554bbaa76} state changed: stopped  result: success
2020-11-20T11:14:06 [I|bac|] Task {label: Actions::Katello::ContentView::Update, id: f484ed33-3d93-4b79-a434-8072dca0bc12, execution_plan_id: 5da530de-3136-412c-8577-83d2871cdfab} state changed: running
2020-11-20T11:14:06 [I|bac|] Task {label: Actions::Katello::ContentView::Update, id: f484ed33-3d93-4b79-a434-8072dca0bc12, execution_plan_id: 5da530de-3136-412c-8577-83d2871cdfab} state changed: stopped  result: success
2020-11-20T11:14:06 [I|bac|] Task {label: Actions::Katello::ContentView::Update, id: f484ed33-3d93-4b79-a434-8072dca0bc12, execution_plan_id: 5da530de-3136-412c-8577-83d2871cdfab} state changed: stopped  result: success
2020-11-20T11:15:24 [I|bac|] Task {label: Actions::Katello::ContentView::Publish, id: 0573458d-8a4f-4224-83ce-3e8c839630c5, execution_plan_id: e77becc8-cfdb-4cb0-8baa-1170676c95a0} state changed: running
2020-11-20T11:24:20 [E|bac|] Error scheduling refresh job. (Katello::Errors::CandlepinError)
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in `rescue in propagate_candlepin_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/propagate_candlepin_errors.rb:18:in `propagate_candlepin_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/propagate_candlepin_errors.rb:13:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:20:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:20:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:20:in `block in finalize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:19:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:23:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:15:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:15:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:19:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:19:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:19:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:19:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:19:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:19:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:40:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:583:in `block in execute_finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:582:in `execute_finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:78:in `run_step'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:63:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:70:in `block in run_in_sequence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:70:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:70:in `all?'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:70:in `run_in_sequence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:59:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:28:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:48:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:48:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:48:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:48:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:48:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:48:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:48:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/common/transaction.rb:10:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director/sequential_manager.rb:27:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director.rb:122:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors.rb:12:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:45:in `work_finished'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:58:in `worker_done'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:23:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:99:in `distribute_jobs'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/pool.rb:52:in `schedule_work'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:48:in `block in feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/core.rb:46:in `feed_pool'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:123:in `handle_work'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:35:in `handle_event'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel.rb:42:in `delayed_event'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/clock.rb:36:in `apply'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/clock.rb:75:in `run_ready_timers'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/clock.rb:57:in `tick'
[ concurrent-ruby ]
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/clock.rb:96:in `block in sleep_to'
[ concurrent-ruby ]

This seems to be coming from candlepin log /var/log/candlepin/candlepin.log

2020-11-20 11:24:20,581 [thread=http-bio-8443-exec-6] [req=e66faac4-6999-4635-b110-d7dbc68bef43, org=, csid=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/environments/04584922e604ece5b217d7a0eb6dd5b5/content
2020-11-20 11:24:20,913 [thread=http-bio-8443-exec-6] [req=e66faac4-6999-4635-b110-d7dbc68bef43, org=OrgName, csid=] ERROR org.candlepin.pinsetter.core.PinsetterKernel - There was a problem scheduling regen_entitlement_cert_of_env119ffd4e-9a62-4c7b-b739-3ed663c7d06c
java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.candlepin.pinsetter.core.PinsetterKernel.scheduleJob(PinsetterKernel.java:405)
        at org.candlepin.pinsetter.core.PinsetterKernel.scheduleSingleJob(PinsetterKernel.java:489)
        at org.candlepin.resteasy.filter.PinsetterAsyncFilter.scheduleJob(PinsetterAsyncFilter.java:99)
        at org.candlepin.resteasy.filter.PinsetterAsyncFilter.postProcess(PinsetterAsyncFilter.java:73)
        at org.jboss.resteasy.core.interception.ContainerResponseFilterRegistry$ContainerResponseFilterFacade.filter(ContainerResponseFilterRegistry.java:47)
        at org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
        at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
        at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
        at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
        at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
        at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at org.candlepin.servlet.filter.EventFilter.doFilter(EventFilter.java:65)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.candlepin.common.filter.LoggingFilter.doFilter(LoggingFilter.java:113)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.candlepin.servlet.filter.CandlepinPersistFilter.doFilter(CandlepinPersistFilter.java:48)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.candlepin.servlet.filter.CandlepinScopeFilter.doFilter(CandlepinScopeFilter.java:68)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.quartz.JobPersistenceException: Couldn't store job: ERROR: cannot execute INSERT in a read-only transaction
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1118)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$2.executeVoid(JobStoreSupport.java:1062)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3703)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3701)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3787)
        at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:93)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1058)
        at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:886)
        at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:249)
        at org.candlepin.pinsetter.tasks.KingpinJob.scheduleJob(KingpinJob.java:227)
        ... 55 common frames omitted
Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute INSERT in a read-only transaction
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
        at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:132)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:384)
        at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.insertJobDetail(StdJDBCDelegate.java:624)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1112)
        ... 64 common frames omitted
2020-11-20 11:24:20,916 [thread=http-bio-8443-exec-6] [req=e66faac4-6999-4635-b110-d7dbc68bef43, org=OrgName, csid=] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=503, content-type="application/json", time=335

Expected outcome: Publish new content view version without errors

Foreman and Proxy versions: 1.24.3

Foreman and Proxy plugin versions: Dynflow: 0.2.4 Pulp: 1.5.0 SSH: 0.2.1 Katello: 3.14.1

Distribution and version: CentOS 7

Other relevant data:

I forgot to add that when the task errors, I can go in there and hit the Resume button and it succeeds with no further problems

Hey @rivermigue,

Thanks for raising this. Did the task succeed upon resuming. Are you still seeing this issue in your environment?

Thanks & Regards,
Samir Jha

hi @sajha, yest it succeeds and it is still happening, I just published a new version for another content view and the same happened.

Could you provide the logs for the individual dynflow task that hangs? What was the bulk action in the logs in the post?

Where would that log be stored at? /var/log/foreman-proxy/smart_proxy_dynflow_core.log

You can go into the paused task in the UI and click on dynflow console in the task details page and get the logs for the paused task from there.

Could you also check the available free disk space on your box. Candlepin behaves weirdly when disk usage grows above 90%.

I will try to reproduce it again, since I had hit the resume button, those tasks are now showing as stopped and successful with no errors inside. There is also 60% of disk space available:

df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 3.8G     0  3.8G   0% /dev
tmpfs                    3.9G   16K  3.9G   1% /dev/shm
tmpfs                    3.9G  197M  3.7G   6% /run
tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root  696G  273G  424G  40% /
/dev/sda1               1014M  217M  798M  22% /boot
tmpfs                    781M     0  781M   0% /run/user/1285805809

rpms are stored in /var

Also, this happened when I tried to push 55 extra repositories at once with 56k packages, it took a bit of time when the publishing task ran the first, I tried to publish an older version with 5k packages and then go back to the 56k packages version, however this time it doesnt throw errors and completes the publishing in about 1 min which is way faster compared to the first time when it threw the errors.