Repo sync failed with: duplicate key value violates unique constraint

Problem:
Syncing AlmaLinux 8 Base repository fails with:
duplicate key value violates unique constraint “rpm_updatecollection_name_update_record_id_6ef33bed_uniq”

Expected outcome:
synced successfully

Foreman and Proxy versions:
foreman - 3.0.1

Foreman and Proxy plugin versions:
katello - 4.2.2

Distribution and version:
CentOS Linux release 7.9.2009

Other relevant data:

2022-06-17T16:04:06 [I|app|0e3fce1b] Completed 200 OK in 39ms (Views: 0.6ms | ActiveRecord: 4.4ms | Allocations: 25621)
2022-06-17T16:04:08 [E|bac|82338e5d] duplicate key value violates unique constraint "rpm_updatecollection_name_update_record_id_6ef33bed_uniq"
 82338e5d | DETAIL:  Key (name, update_record_id)=(almalinux-8-for-x86_64-appstream-rpms__8_1_subversion_0, ed7f0e58-f17e-49b3-85ad-b26da2e9c8a9) already exists.
 82338e5d |  (Katello::Errors::Pulp3Error)
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:139:in `poll_external_task'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:22:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/cancellable.rb:14:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:17:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/world.rb:31:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:571:in `block (2 levels) in execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in `catch'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in `block in execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `block in with_error_handling'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `catch'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `with_error_handling'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:565:in `execute_run'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:286:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/director.rb:94:in `execute'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors.rb:18:in `run_user_code'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 82338e5d | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 82338e5d | [ sidekiq ]
 82338e5d | [ concurrent-ruby ]

I could resolve it by yum update and foreman-maintain service restart.
After that, I could sync without issues.

Updates resulted in:

---> Package foreman.noarch 0:3.0.1-1.el7 will be updated
---> Package foreman.noarch 0:3.0.2-1.el7 will be an update

---> Package python3-pulp-rpm.noarch 0:3.14.8-1.el7 will be updated
---> Package python3-pulp-rpm.noarch 0:3.14.14-1.el7 will be an update

---> Package python3-pulpcore.noarch 0:3.14.9-1.el7 will be updated
---> Package python3-pulpcore.noarch 0:3.14.15-1.el7 will be an update

It‘s better to always run foreman-installer after updating. There may be some migrations pending.

# yum update
# foreman-maintain service stop
# foreman-installer
2 Likes

It’s likely there was a bug fix in that large pulp-rpm or pulpcore version jump :slight_smile:

+1 to @gvde saying to run the installer after updating, it’s the only way to run pulpcore migrations besides running them manually.

I did same steps but still face issue. How could we further investigate this issue?
I use Foreman 2.3.5

Are you syncing multiple repositories at once or perhaps the same repository multiple times? I can see how perhaps that might trigger this issue if Alma Linux has the same errata in both repos.

Foreman/Katello 3.3/4.5.
Also got this blocking problem when syncing AlmaLinux 8 baseos and appstream repos:

duplicate key value violates unique constraint "rpm_updatecollection_name_update_record_id_6ef33bed_uniq"
DETAIL:  Key (name, update_record_id)=(almalinux-8-for-x86_64-appstream-rpms__8_1_subversion_0, 9caae809-4ea1-493b-8c61-6761a38410de) already exists.

Also tried a “Complete Sync” with same result. Super annoying since no packages are imported to the repo due to this.
Repo settings:
Download Policy: Immediate
Mirroring Policy: Additive
Retain package versions: 5

Looking in /var/lib/pgsql/data/log/postgresql-Fri.log is it flooded with the same error message for what it seems every package it tries to add.
Small snippet:

2022-07-29 16:24:28 UTC ERROR:  duplicate key value violates unique constraint "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq"
2022-07-29 16:24:28 UTC DETAIL:  Key (name, stream, version, context, arch)=(swig, 3.0, 8030020210105114017, 30b713e6, x86_64) already exists.
2022-07-29 16:24:28 UTC STATEMENT:  INSERT INTO "rpm_modulemd" ("content_ptr_id", "name", "stream", "version", "context", "arch", "static_context", "dependencies", "artifacts") VALUES ('581670ec-5022-40b0-8870-ea518b710ea4'::uuid, 'swig', '3.0', '8030020210105114017', '30b713e6', 'x86_64', false, '[{"platform": ["el8"]}]', '["swig-0:3.0.12-19.module_el8.3.0+6167+838326ab.src", "swig-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64", "swig-debuginfo-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64", "swig-debugsource-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64", "swig-doc-0:3.0.12-19.module_el8.3.0+6167+838326ab.noarch", "swig-gdb-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64"]')
2022-07-29 16:24:28 UTC ERROR:  duplicate key value violates unique constraint "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq"
2022-07-29 16:24:28 UTC DETAIL:  Key (name, stream, version, context, arch)=(swig, 4.0, 8040020210408185259, 9f9e2e7e, x86_64) already exists.
2022-07-29 16:24:28 UTC STATEMENT:  INSERT INTO "rpm_modulemd" ("content_ptr_id", "name", "stream", "version", "context", "arch", "static_context", "dependencies", "artifacts") VALUES ('2168caa8-3d6b-48fd-834f-a5f4cb289d6f'::uuid, 'swig', '4.0', '8040020210408185259', '9f9e2e7e', 'x86_64', false, '[{"platform": ["el8"]}]', '["swig-0:4.0.2-3.module_el8.4.0+2100+4bdcb5c6.src", "swig-0:4.0.2-3.module_el8.4.0+2100+4bdcb5c6.x86_64", "swig-debuginfo-0:4.0.2-3.module_el8.4.0+2100+4bdcb5c6.x86_64", "swig-debugsource-0:4.0.2-3.module_el8.4.0+2100+4bdcb5c6.x86_64", "swig-doc-0:4.0.2-3.module_el8.4.0+2100+4bdcb5c6.noarch", "swig-gdb-0:4.0.2-3.module_el8.4.0+2100+4bdcb5c6.x86_64"]')
2022-07-29 16:24:28 UTC ERROR:  duplicate key value violates unique constraint "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq"
2022-07-29 16:24:28 UTC DETAIL:  Key (name, stream, version, context, arch)=(varnish, 6, 8050020220204155301, c5368500, x86_64) already exists.
2022-07-29 16:24:28 UTC STATEMENT:  INSERT INTO "rpm_modulemd" ("content_ptr_id", "name", "stream", "version", "context", "arch", "static_context", "dependencies", "artifacts") VALUES ('39136541-d091-48ae-a485-8668c7220421'::uuid, 'varnish', '6', '8050020220204155301', 'c5368500', 'x86_64', false, '[{"platform": ["el8"]}]', '["varnish-0:6.0.8-1.module_el8.5.0+2620+03a0c2cc.1.src", "varnish-0:6.0.8-1.module_el8.5.0+2620+03a0c2cc.1.x86_64", "varnish-devel-0:6.0.8-1.module_el8.5.0+2620+03a0c2cc.1.x86_64", "varnish-docs-0:6.0.8-1.module_el8.5.0+2620+03a0c2cc.1.x86_64", "varnish-modules-0:0.15.0-6.module_el8.5.0+2620+03a0c2cc.src", "varnish-modules-0:0.15.0-6.module_el8.5.0+2620+03a0c2cc.x86_64", "varnish-modules-debuginfo-0:0.15.0-6.module_el8.5.0+2620+03a0c2cc.x86_64", "varnish-modules-debugsource-0:0.15.0-6.module_el8.5.0+2620+03a0c2cc.x86_64"]')
2022-07-29 16:24:28 UTC ERROR:  duplicate key value violates unique constraint "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq"
2022-07-29 16:24:28 UTC DETAIL:  Key (name, stream, version, context, arch)=(virt, rhel, 8060020220510122810, 9edba152, x86_64) already exists.

Searching for one rpm seen in the log:

2022-07-29 16:24:28 UTC STATEMENT:  INSERT INTO "rpm_modulemd" ("content_ptr_id", "name", "stream", "version", "context", "arch", "static_context", "dependencies", "artifacts") VALUES ('581670ec-5022-40b0-8870-ea518b710ea4'::uuid, 'swig', '3.0', '8030020210105114017', '30b713e6', 'x86_64', false, '[{"platform": ["el8"]}]', '["swig-0:3.0.12-19.module_el8.3.0+6167+838326ab.src", "swig-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64", "swig-debuginfo-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64", "swig-debugsource-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64", "swig-doc-0:3.0.12-19.module_el8.3.0+6167+838326ab.noarch", "swig-gdb-0:3.0.12-19.module_el8.3.0+6167+838326ab.x86_64"]')

in the repo shows of course that package is already in the repo:

#### File Information
Size 1.53 MB (1603532 Bytes)
Filename swig-3.0.12-19.module_el8.3.0+6167+838326ab.x86_64.rpm
Checksum 55eb69b5d89170fcd7be10903310d49357818b0eb2d9acbb605df5815b223bd9
Checksum Type sha256

#### Build Information
Source RPM swig-3.0.12-19.module_el8.3.0+6167+838326ab.src.rpm
Build Host build.cloudlinux.com
Build Time January 05, 2021 at 01:03 PM

Not sure how that can be a problem though. I assume it will always find packages that already exist in the repo that it should not add. I guess I can remove the repo and build it up again from scratch but since I am not the only one with this error, might be worth debugging it.

So looking more on the postgres log I see 136 ERROR entries for one sync with these 3 types:

130 of these:
ERROR: duplicate key value violates unique constraint “rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq”

5 of these:
ERROR: duplicate key value violates unique constraint “rpm_updaterecord_digest_key”

One line with this one:
ERROR: duplicate key value violates unique constraint “rpm_updatecollection_name_update_record_id_6ef33bed_uniq”

Alma Linux has been doing some seriously weird stuff with their Errata. I’ll try to dig into exactly why soon but their repos seem to be the only ones having all these issues…

Ended up changing the mirror policy for these repos from “additive” to “content only”.
After the change the sync was successful. I then changed it back to “additive” and then performed another successful sync.

That’s good to know, I’ll have to keep that workaround in mind. Maybe there was a temporary mistake in the metadata that got resolved but “additive” wasn’t purging the bad metadata.

This is all just speculation of course, glad it’s working now.