Red Hat repository fails to sync (Pulp3Error)

Problem:
Trying to sync Red Hat repository fails… I’ve tried disabling and re-syncing but I get the same error.

Katello 3.16.1
Pulp Server 2.21.3

Foreman and Proxy plugin versions:
Foreman 2.1.4
Foreman Proxy 2.1.4

Distribution and version:
RHEL 7.9

Logs
Katello::Errors::Pulp3Error: duplicate key value violates unique constraint “core_repositoryversion_repository_id_number_3c54ce50_uniq”
DETAIL: Key (repository_id, number)=(412adda7-d6dd-4e52-a278-3dfc735532f0, 1) already exists.

{“pulp_tasks”=>
[{“pulp_href”=>"/pulp/api/v3/tasks/4072bfe2-ce52-40ca-a3b8-74532403c226/",
“pulp_created”=>“2020-10-22T22:17:57.672+00:00”,
“state”=>“failed”,
“name”=>“pulp_rpm.app.tasks.synchronizing.synchronize”,
“started_at”=>“2020-10-22T22:17:57.780+00:00”,
“finished_at”=>“2020-10-22T22:17:58.209+00:00”,
“error”=>
{“traceback”=>
" File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 883, in perform_job\n" +
" rv = job.perform()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 645, in perform\n" +
" self._result = self._execute()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 651, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 264, in synchronize\n" +
" dv.create()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 140, in create\n" +
" with self.repository.new_version() as new_version:\n" +
" File “/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py”, line 105, in new_version\n" +
" version.save()\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/base.py”, line 741, in save\n" +
" force_update=force_update, update_fields=update_fields)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/base.py”, line 779, in save_base\n" +
" force_update, using, update_fields,\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/base.py”, line 870, in _save_table\n" +
" result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/base.py”, line 908, in _do_insert\n" +
" using=using, raw=raw)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/manager.py”, line 82, in manager_method\n" +
" return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 1186, in _insert\n" +
" return query.get_compiler(using=using).execute_sql(return_id)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1375, in execute_sql\n" +
" cursor.execute(sql, params)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute\n" +
" return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers\n" +
" return executor(sql, params, many, context)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute\n" +
" return self.cursor.execute(sql, params)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit\n" +
" raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute\n" +
" return self.cursor.execute(sql, params)\n",
“description”=>
“duplicate key value violates unique constraint “core_repositoryversion_repository_id_number_3c54ce50_uniq”\n” +
“DETAIL: Key (repository_id, number)=(412adda7-d6dd-4e52-a278-3dfc735532f0, 1) already exists.\n”},
“worker”=>"/pulp/api/v3/workers/3fbbd5dc-a5d4-4016-a843-21f0e97a1650/",
“child_tasks”=>,
“progress_reports”=>,
“created_resources”=>,
“reserved_resources_record”=>
["/pulp/api/v3/repositories/rpm/rpm/412adda7-d6dd-4e52-a278-3dfc735532f0/",
“/pulp/api/v3/remotes/rpm/rpm/6bac267f-5179-4a7d-8ca9-956e5aa978df/”]}],
“create_version”=>true,
“task_groups”=>,
“poll_attempts”=>{“total”=>1, “failed”=>1}}

Also same issue when I try to sync “Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server”
Synchronize repository ‘Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server’; product ‘Red Hat Enterprise Linux Server’;

Katello::Errors::Pulp3Error: duplicate key value violates unique constraint “core_repositoryversion_repository_id_number_3c54ce50_uniq”
DETAIL: Key (repository_id, number)=(86be12d0-1509-42cc-a9da-681cc55c0bc0, 2) already exists.

Something I need to fix / delete in Pulp database?

I believe you are running into this issue: https://pulp.plan.io/issues/6463

You could either try applying the patch (see https://github.com/pulp/pulpcore/pull/902) or you could try removing the incomplete RepositoryVersions by hand. Here’s the command that works in a Pulp environment (not sure about the manager command in a Katello environment):

pulpcore-manager shell -c "from pulpcore.app.models import RepositoryVersion; RepositoryVersion.objects.filter(complete=False).delete()"
1 Like

If this is a production system, I’d probably advise against running the command. If you do run it, be sure you backup first and stop Katello/Pulp from running.

Justin filed a backport request though so it may even be worth waiting for it: https://pulp.plan.io/issues/7737.

I believe i just ran into this same error after upgrading from 3.16 to 3.17:

Katello::Errors::Pulp3Error: duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"
DETAIL:  Key (repository_id, content_id, version_removed_id)=(5cf832a8-c96a-4336-94f2-1617e2f6f3fa, 0b09dd09-de5e-45a4-9807-61961c29b685, 6f8416a0-8cc2-4a85-b458-5634cad2a6d9) already exists.

I reviewed the github commit and tried to manually apply fix, but got a pulp 503 error, so i rolled back my VM. These are the steps i had done:

Can you elaborate on what you mean by removing conflict by hand with pulpcore-manager? Wasnt sure how to use that command and im not getting fruitful results.