Repo sync failed with: duplicate key value violates unique constraint

Problem:
Syncing AlmaLinux 8 Base repository fails with:
duplicate key value violates unique constraint “rpm_updatecollection_name_update_record_id_6ef33bed_uniq”

Expected outcome:
synced successfully

Foreman and Proxy versions:
foreman - 3.0.1

Foreman and Proxy plugin versions:
katello - 4.2.2

Distribution and version:
CentOS Linux release 7.9.2009

Other relevant data:

2022-06-17T16:04:06 [I|app|0e3fce1b] Completed 200 OK in 39ms (Views: 0.6ms | ActiveRecord: 4.4ms | Allocations: 25621)
2022-06-17T16:04:08 [E|bac|82338e5d] duplicate key value violates unique constraint "rpm_updatecollection_name_update_record_id_6ef33bed_uniq"
 82338e5d | DETAIL:  Key (name, update_record_id)=(almalinux-8-for-x86_64-appstream-rpms__8_1_subversion_0, ed7f0e58-f17e-49b3-85ad-b26da2e9c8a9) already exists.
 82338e5d |  (Katello::Errors::Pulp3Error)
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:139:in `poll_external_task'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:22:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/cancellable.rb:14:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:17:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/world.rb:31:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:571:in `block (2 levels) in execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in `catch'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in `block in execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `block in with_error_handling'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `catch'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `with_error_handling'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:565:in `execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:286:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/director.rb:94:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors.rb:18:in `run_user_code'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 82338e5d | [ sidekiq ]
 82338e5d | [ concurrent-ruby ]

I could resolve it by yum update and foreman-maintain service restart.
After that, I could sync without issues.

Updates resulted in:

---> Package foreman.noarch 0:3.0.1-1.el7 will be updated
---> Package foreman.noarch 0:3.0.2-1.el7 will be an update

---> Package python3-pulp-rpm.noarch 0:3.14.8-1.el7 will be updated
---> Package python3-pulp-rpm.noarch 0:3.14.14-1.el7 will be an update

---> Package python3-pulpcore.noarch 0:3.14.9-1.el7 will be updated
---> Package python3-pulpcore.noarch 0:3.14.15-1.el7 will be an update

It‘s better to always run foreman-installer after updating. There may be some migrations pending.

# yum update
# foreman-maintain service stop
# foreman-installer
2 Likes

It’s likely there was a bug fix in that large pulp-rpm or pulpcore version jump :slight_smile:

+1 to @gvde saying to run the installer after updating, it’s the only way to run pulpcore migrations besides running them manually.

I did same steps but still face issue. How could we further investigate this issue?
I use Foreman 2.3.5