The repositorys publication is missing. Please run a complete sync

Problem:
After upgrade to Foreman 3.16 / Katello 4.18 Publish new Content view end with: The repository’s publication is missing. Please run a ‘complete sync’ … Repeated for every repository in Content view. Task ends with error in paused state. In dynflow console failed step is

Actions::Pulp3::Repository::RefreshDistribution (error).

This hapens for all existing contenviews and also for newly created. I tried also create new product, with new repository and new content view with same result. If I skip failed step in dynflow, task can be resumed, but created content view is empty.

Expected outcome:
Created ContentView

Foreman and Proxy versions:
Foreman 3.16 / Katello 4.18

Distribution and version:
Rocky Linux 9.6

Other relevant data:
Error log from Dynflow

---
- "/usr/share/gems/gems/katello-4.18.0/app/services/katello/pulp3/repository.rb:563:in
  `fail_missing_publication'"
- "/usr/share/gems/gems/katello-4.18.0/app/services/katello/pulp3/repository.rb:301:in
  `update_distribution'"
- "/usr/share/gems/gems/katello-4.18.0/app/services/katello/pulp3/repository.rb:261:in
  `refresh_distributions'"
- "/usr/share/gems/gems/katello-4.18.0/app/lib/actions/pulp3/repository/refresh_distribution.rb:26:in
  `invoke_external_task'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action/polling.rb:84:in `initiate_external_action'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action/polling.rb:19:in `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action/cancellable.rb:15:in `run'"
- "/usr/share/gems/gems/katello-4.18.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:590:in `block (3 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:33:in `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/katello-4.18.0/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/usr/share/gems/gems/katello-4.18.0/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/usr/share/gems/gems/katello-4.18.0/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/usr/share/gems/gems/katello-4.18.0/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/usr/share/gems/gems/katello-4.18.0/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/usr/share/gems/gems/katello-4.18.0/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in
  `wrap'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action/progress.rb:15:in `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:33:in `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/usr/share/gems/gems/foreman-tasks-11.0.4/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/world.rb:31:in `execute'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:589:in `block (2 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:588:in `catch'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:588:in `block in execute_run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:491:in `block in with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:491:in `catch'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:491:in `with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:583:in `execute_run'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:304:in `execute'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract.rb:168:in
  `with_meta_calculation'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/director.rb:70:in `execute'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in
  `block (2 levels) in perform'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors.rb:18:in `run_user_code'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in
  `block in perform'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in
  `with_telemetry'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `perform'"
- "/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/serialization.rb:28:in
  `perform'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels)
  in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'"
- "/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block
  in create_with_logging_context'"

Same here

Hi @dano.pogac ,

Please provide the information requested in this similar thread: The repository's publication is missing - #6 by iballou

Seeing this error alone isn’t enough to debug the failure. It’s likely that one of the previous actions in the orchestration chain caused this particular error.

Also, any back traces in /var/log/messages from Pulp (if applicable) would be helpful.