Katello 4.1 content proxy sync errors

I have just upgrade my main foreman server from 3.18.3 via 4.0 to 4.1. As I had a lot of problems with upgrading my content proxy (see Katello 4.0 content proxy upgrade disaster) I have installed a new server with the 4.1 content proxy.

My initial optimized sync of the new content proxy shows a couple of error messages:

deadlock detected
DETAIL:  Process 12693 waits for ShareLock on transaction 27951; blocked by process 12680.
Process 12680 waits for ShareLock on transaction 27986; blocked by process 12693.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (10,1) in relation "rpm_package_pkgId_key"
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 0d20a02b-d3a6-42cd-b333-d99e91d0e482) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 0d20a02b-d3a6-42cd-b333-d99e91d0e482) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 0d20a02b-d3a6-42cd-b333-d99e91d0e482) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(extras, extras, extras, variant, Packages, ca24473d-b2d7-4535-b2a2-f76db58868ac) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, e1520ccf-aa2c-42af-bcee-c785ed7feba4) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, e1520ccf-aa2c-42af-bcee-c785ed7feba4) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, e1520ccf-aa2c-42af-bcee-c785ed7feba4) already exists.
deadlock detected
DETAIL:  Process 16562 waits for ShareLock on transaction 44181; blocked by process 16589.
Process 16589 waits for ShareLock on transaction 43893; blocked by process 16562.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (871,1) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 16576 waits for ShareLock on transaction 43893; blocked by process 16562.
Process 16562 waits for ShareLock on transaction 44085; blocked by process 16576.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (863,2) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 16614 waits for ShareLock on transaction 44181; blocked by process 16589.
Process 16589 waits for ShareLock on transaction 44312; blocked by process 16614.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (914,3) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 17857 waits for ShareLock on transaction 50756; blocked by process 17902.
Process 17902 waits for ShareLock on transaction 50064; blocked by process 17857.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1503,3) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 17872 waits for ShareLock on transaction 50064; blocked by process 17857.
Process 17857 waits for ShareLock on transaction 50339; blocked by process 17872.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1482,2) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 17902 waits for ShareLock on transaction 50539; blocked by process 17885.
Process 17885 waits for ShareLock on transaction 50756; blocked by process 17902.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1637,4) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 21733 waits for ShareLock on transaction 62564; blocked by process 21742.
Process 21742 waits for ShareLock on transaction 62458; blocked by process 21733.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (987,4) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 21742 waits for ShareLock on transaction 62719; blocked by process 21784.
Process 21784 waits for ShareLock on transaction 62564; blocked by process 21742.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1046,2) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 21784 waits for ShareLock on transaction 62647; blocked by process 21760.
Process 21760 waits for ShareLock on transaction 62719; blocked by process 21784.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1156,2) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 21810 waits for ShareLock on transaction 62744; blocked by process 21796.
Process 21796 waits for ShareLock on transaction 62870; blocked by process 21810.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1058,2) in relation "rpm_package_pkgId_key"
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 49e4d7b9-e692-4336-82e6-8b2ba84656b8) already exists.
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 49e4d7b9-e692-4336-82e6-8b2ba84656b8) already exists.
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 49e4d7b9-e692-4336-82e6-8b2ba84656b8) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(PowerTools, PowerTools, PowerTools, variant, Packages, 6900b8f3-6f5f-4460-9f9f-f30d1213c8ff) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(PowerTools, PowerTools, PowerTools, variant, Packages, 6900b8f3-6f5f-4460-9f9f-f30d1213c8ff) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(PowerTools, PowerTools, PowerTools, variant, Packages, 6900b8f3-6f5f-4460-9f9f-f30d1213c8ff) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(AppStream, AppStream, AppStream, variant, Packages, 862cdb92-f315-4cd7-9dd7-5014eecc8abe) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(AppStream, AppStream, AppStream, variant, Packages, 862cdb92-f315-4cd7-9dd7-5014eecc8abe) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(AppStream, AppStream, AppStream, variant, Packages, 862cdb92-f315-4cd7-9dd7-5014eecc8abe) already exists.

While I can not help here, I just wanted to say Thank you for all the testing recently!

It all got worse and I also wasn’t able to publish one of main most important CVs anymore. I have got the same error as in Unable to publish new version of content view with filter.

The publish task paused. I could resume and it would take a couple of minutes until it paused again, with slightly different numbers, always sticking at 26%. Looking into the pulpcore database table core_repositorycontent it seems that there are a lot of these cases, so I eventually tried to cancel that task, but in vain. Whatever I have tried, it always kept some locks which prevented me from publishing a new version or removing that broken version…

In the end, it looked like a completely dead end to me, again, and I am back again at 3.18.3. So far I haven’t managed to get this server updated to 4.0 or 4.1 so that I can at least do the standard things of syncing, publishing, promoting and delivering content to my clients, accepting the broken el8 module applicability and the extremely slow publish of CVs with filters…

1 Like

Another attempt, same output for the initial smart proxy sync:

duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 722bd1ee-2634-49aa-8e88-33c22401c231) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 722bd1ee-2634-49aa-8e88-33c22401c231) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 722bd1ee-2634-49aa-8e88-33c22401c231) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(extras, extras, extras, variant, Packages, 4131d139-2c0d-4ec6-b710-c736c71f7607) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8) already exists.
deadlock detected
DETAIL:  Process 29839 waits for ShareLock on transaction 42277; blocked by process 29878.
Process 29878 waits for ShareLock on transaction 42195; blocked by process 29839.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1586,5) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 29865 waits for ShareLock on transaction 42277; blocked by process 29878.
Process 29878 waits for ShareLock on transaction 42262; blocked by process 29865.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (2145,3) in relation "rpm_package_pkgId_key"
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 7dcb382f-7da3-40c6-81f4-fbf6e69e3618) already exists.
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 7dcb382f-7da3-40c6-81f4-fbf6e69e3618) already exists.
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 7dcb382f-7da3-40c6-81f4-fbf6e69e3618) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(PowerTools, PowerTools, PowerTools, variant, Packages, 5e530299-2465-4206-85ec-8bd46eeca432) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(PowerTools, PowerTools, PowerTools, variant, Packages, 5e530299-2465-4206-85ec-8bd46eeca432) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(AppStream, AppStream, AppStream, variant, Packages, b46c3a77-ae7f-46d7-92bc-c44f61d030c6) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(AppStream, AppStream, AppStream, variant, Packages, b46c3a77-ae7f-46d7-92bc-c44f61d030c6) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(AppStream, AppStream, AppStream, variant, Packages, b46c3a77-ae7f-46d7-92bc-c44f61d030c6) already exists.

A publish and promote of my centos7 and centos8 CVs. Then I start another optimized sync which still shows some errors:

insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(7db11963-4adf-49df-b276-d66f6f1c24ce) is not present in table "core_repositoryversion".
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(c3fd26f1-7dc3-465a-bf1a-21ca479fb5c1) is not present in table "core_repositoryversion".
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(2505b1f7-cc8a-489d-aaab-911a30910319) is not present in table "core_repositoryversion".

Now, the big question is: it is generally working okay or is the whole thing broken and is accumulating problems??

Also, the old pulp2 content proxy synchronized automatically after I have promoted a cv into the lifecycle environment on the content proxy. The new pulp3 content proxy seems to need manual intervention to get it synchronized?

You may need to wait for Katello 4.1.1 to come out.

I’ve been seeing some mentions of incorporating Pulp 3.14 into the Katello 4.1.1 release which should fix a number of Pulp3 issues that have been cropping up. Whether it would fix this specific issue, I’m not sure - but some of the ones I’ve run into (like not being able to synchronize repositories if SRPMs are to be ignored) are slated to be fixed in that release.

I am having a closer look at those sql errors, checking the postgresql logs and database content:

First three errors:

duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 722bd1ee-2634-49aa-8e88-33c22401c231) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 722bd1ee-2634-49aa-8e88-33c22401c231) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(plus, plus, plus, variant, Packages, 722bd1ee-2634-49aa-8e88-33c22401c231) already exists.

in the database:

pulpcore=# select * from rpm_variant where (variant_id, uid, name, type, packages, distribution_tree_id)=('plus', 'plus', 'plus', 'variant', 'Packages', '722bd1ee-2634-49aa-8e88-33c22401c231');
               pulp_id                |         pulp_created          |      pulp_last_updated       | variant_id | uid  | name |  type   | packages | source_packa
ges | source_repository | debug_packages | debug_repository | identity |         distribution_tree_id         | repository_id 
--------------------------------------+-------------------------------+------------------------------+------------+------+------+---------+----------+-------------
----+-------------------+----------------+------------------+----------+--------------------------------------+---------------
 d58fd024-6762-4b4e-b6f8-eccf11dbdeea | 2021-07-07 16:27:18.836032+02 | 2021-07-07 16:27:18.83605+02 | plus       | plus | plus | variant | Packages |             
    |                   |                |                  |          | 722bd1ee-2634-49aa-8e88-33c22401c231 | 
(1 row)

pulpcore=# select * from rpm_distributiontree where content_ptr_id = '722bd1ee-2634-49aa-8e88-33c22401c231';
            content_ptr_id            | header_version | release_name | release_short | release_version | release_is_layered | base_product_name | base_product_sho
rt | base_product_version |  arch  | build_timestamp | instimage | mainimage | discnum | totaldiscs 
--------------------------------------+----------------+--------------+---------------+-----------------+--------------------+-------------------+-----------------
---+----------------------+--------+-----------------+-----------+-----------+---------+------------
 722bd1ee-2634-49aa-8e88-33c22401c231 | 1.2            | CentOS-Plus  | Plus          | 8               | f                  |                   |                 
   |                      | x86_64 |      1611948083 |           |           |         |           
(1 row)

failed statements:

2021-07-07 16:27:19 CEST STATEMENT:  INSERT INTO "rpm_variant" ("pulp_id", "pulp_created", "pulp_last_updated", "variant_id", "uid", "name", "type", "packages", "source_packages", "source_repository", "debug_packages", "debug_repository", "identity", "distribution_tree_id", "repository_id") VALUES ('890eafc6-4d11-4444-b509-668da028bbc1'::uuid, '2021-07-07T14:27:19.132590+00:00'::timestamptz, '2021-07-07T14:27:19.132607+00:00'::timestamptz, 'plus', 'plus', 'plus', 'variant', 'Packages', NULL, NULL, NULL, NULL, NULL, '722bd1ee-2634-49aa-8e88-33c22401c231'::uuid, NULL) RETURNING "rpm_variant"."pulp_id"

2021-07-07 16:27:19 CEST STATEMENT:  INSERT INTO "rpm_variant" ("pulp_id", "pulp_created", "pulp_last_updated", "variant_id", "uid", "name", "type", "packages", "source_packages", "source_repository", "debug_packages", "debug_repository", "identity", "distribution_tree_id", "repository_id") VALUES ('8199f985-81d0-4871-a7a1-50621d7c0643'::uuid, '2021-07-07T14:27:19.570542+00:00'::timestamptz, '2021-07-07T14:27:19.570558+00:00'::timestamptz, 'plus', 'plus', 'plus', 'variant', 'Packages', NULL, NULL, NULL, NULL, NULL, '722bd1ee-2634-49aa-8e88-33c22401c231'::uuid, NULL) RETURNING "rpm_variant"."pulp_id"

2021-07-07 16:27:19 CEST STATEMENT:  INSERT INTO "rpm_variant" ("pulp_id", "pulp_created", "pulp_last_updated", "variant_id", "uid", "name", "type", "packages", "source_packages", "source_repository", "debug_packages", "debug_repository", "identity", "distribution_tree_id", "repository_id") VALUES ('d0ca8175-3b7c-4617-98ea-d99cd64a6e27'::uuid, '2021-07-07T14:27:19.587468+00:00'::timestamptz, '2021-07-07T14:27:19.587483+00:00'::timestamptz, 'plus', 'plus', 'plus', 'variant', 'Packages', NULL, NULL, NULL, NULL, NULL, '722bd1ee-2634-49aa-8e88-33c22401c231'::uuid, NULL) RETURNING "rpm_variant"."pulp_id"

next error messages:

duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(extras, extras, extras, variant, Packages, 4131d139-2c0d-4ec6-b710-c736c71f7607) already exists.

database:

pulpcore=# select * from rpm_variant where (variant_id, uid, name, type, packages, distribution_tree_id)=('extras', 'extras', 'extras', 'variant', 'Packages', '4131d139-2c0d-4ec6-b710-c736c71f7607');
               pulp_id                |         pulp_created          |      pulp_last_updated       | variant_id |  uid   |  name  |  type   | packages | source_p
ackages | source_repository | debug_packages | debug_repository | identity |         distribution_tree_id         | repository_id 
--------------------------------------+-------------------------------+------------------------------+------------+--------+--------+---------+----------+---------
--------+-------------------+----------------+------------------+----------+--------------------------------------+---------------
 b3df46e5-0b2e-4528-912e-012fb7c1ae58 | 2021-07-07 16:27:19.326955+02 | 2021-07-07 16:27:19.32697+02 | extras     | extras | extras | variant | Packages |         
        |                   |                |                  |          | 4131d139-2c0d-4ec6-b710-c736c71f7607 | 
(1 row)

pulpcore=# select * from rpm_distributiontree where content_ptr_id = '4131d139-2c0d-4ec6-b710-c736c71f7607';
            content_ptr_id            | header_version | release_name  | release_short | release_version | release_is_layered | base_product_name | base_product_sh
ort | base_product_version |  arch  | build_timestamp | instimage | mainimage | discnum | totaldiscs 
--------------------------------------+----------------+---------------+---------------+-----------------+--------------------+-------------------+----------------
----+----------------------+--------+-----------------+-----------+-----------+---------+------------
 4131d139-2c0d-4ec6-b710-c736c71f7607 | 1.2            | CentOS-Extras | Extras        | 8               | f                  |                   |                
    |                      | x86_64 |      1605188561 |           |           |         |           
(1 row)

failed statement:

2021-07-07 16:27:19 CEST STATEMENT:  INSERT INTO "rpm_variant" ("pulp_id", "pulp_created", "pulp_last_updated", "variant_id", "uid", "name", "type", "packages", "source_packages", "source_repository", 
"debug_packages", "debug_repository", "identity", "distribution_tree_id", "repository_id") VALUES ('716d0784-3964-46cd-bc93-5a1057be34d9'::uuid, '2021-07-07T14:27:19.493204+00:00'::timestamptz, '2021-0
7-07T14:27:19.493220+00:00'::timestamptz, 'extras', 'extras', 'extras', 'variant', 'Packages', NULL, NULL, NULL, NULL, NULL, '4131d139-2c0d-4ec6-b710-c736c71f7607'::uuid, NULL) RETURNING "rpm_variant".
"pulp_id"

next errors:

duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(HighAvailability, HighAvailability, High Availability, variant, Packages, 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8) already exists.

database:

pulpcore=# select * from rpm_variant where  (variant_id, uid, name, type, packages, distribution_tree_id)=('HighAvailability', 'HighAvailability', 'High Availability', 'variant', 'Packages', '2b26cc94-5bf7-47a2-b110-5ac208b2cbe8');
               pulp_id                |         pulp_created          |       pulp_last_updated       |    variant_id    |       uid        |       name        |  
type   | packages | source_packages | source_repository | debug_packages | debug_repository | identity |         distribution_tree_id         | repository_id 
--------------------------------------+-------------------------------+-------------------------------+------------------+------------------+-------------------+--
-------+----------+-----------------+-------------------+----------------+------------------+----------+--------------------------------------+---------------
 6235152d-b8cc-47c8-a835-f4952fd18b42 | 2021-07-07 16:27:23.062999+02 | 2021-07-07 16:27:23.063014+02 | HighAvailability | HighAvailability | High Availability | v
ariant | Packages |                 |                   |                |                  |          | 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8 | 
(1 row)

pulpcore=# select * from rpm_distributiontree where content_ptr_id = '2b26cc94-5bf7-47a2-b110-5ac208b2cbe8';
            content_ptr_id            | header_version | release_name | release_short | release_version | release_is_layered | base_product_name | base_product_sho
rt | base_product_version |  arch  | build_timestamp | instimage | mainimage | discnum | totaldiscs 
--------------------------------------+----------------+--------------+---------------+-----------------+--------------------+-------------------+-----------------
---+----------------------+--------+-----------------+-----------+-----------+---------+------------
 2b26cc94-5bf7-47a2-b110-5ac208b2cbe8 | 1.2            | CentOS Linux | CentOS        | 8               | f                  |                   |                 
   |                      | x86_64 |      1622579918 |           |           |         |           
(1 row)

failed statements:

2021-07-07 16:27:24 CEST STATEMENT:  INSERT INTO "rpm_variant" ("pulp_id", "pulp_created", "pulp_last_updated", "variant_id", "uid", "name", "type", "packages", "source_packages", "source_repository", "debug_packages", "debug_repository", "identity", "distribution_tree_id", "repository_id") VALUES ('dc83b6ec-0664-497e-9a16-8f3332ec5c98'::uuid, '2021-07-07T14:27:24.507285+00:00'::timestamptz, '2021-07-07T14:27:24.507302+00:00'::timestamptz, 'HighAvailability', 'HighAvailability', 'High Availability', 'variant', 'Packages', NULL, NULL, NULL, NULL, NULL, '2b26cc94-5bf7-47a2-b110-5ac208b2cbe8'::uuid, NULL) RETURNING "rpm_variant"."pulp_id"

2021-07-07 16:27:24 CEST STATEMENT:  INSERT INTO "rpm_variant" ("pulp_id", "pulp_created", "pulp_last_updated", "variant_id", "uid", "name", "type", "packages", "source_packages", "source_repository", "debug_packages", "debug_repository", "identity", "distribution_tree_id", "repository_id") VALUES ('bff75e97-4bc4-471c-bf58-bbf2f248f1c8'::uuid, '2021-07-07T14:27:24.536791+00:00'::timestamptz, '2021-07-07T14:27:24.536809+00:00'::timestamptz, 'HighAvailability', 'HighAvailability', 'High Availability', 'variant', 'Packages', NULL, NULL, NULL, NULL, NULL, '2b26cc94-5bf7-47a2-b110-5ac208b2cbe8'::uuid, NULL) RETURNING "rpm_variant"."pulp_id"

2021-07-07 16:27:24 CEST STATEMENT:  INSERT INTO "rpm_variant" ("pulp_id", "pulp_created", "pulp_last_updated", "variant_id", "uid", "name", "type", "packages", "source_packages", "source_repository", "debug_packages", "debug_repository", "identity", "distribution_tree_id", "repository_id") VALUES ('c284cdc1-c021-49eb-8770-12127594763e'::uuid, '2021-07-07T14:27:24.547098+00:00'::timestamptz, '2021-07-07T14:27:24.547113+00:00'::timestamptz, 'HighAvailability', 'HighAvailability', 'High Availability', 'variant', 'Packages', NULL, NULL, NULL, NULL, NULL, '2b26cc94-5bf7-47a2-b110-5ac208b2cbe8'::uuid, NULL) RETURNING "rpm_variant"."pulp_id"

I guess that should be enough. The other variant errors are regarding PowerTools and AppStream. I have CentOS 8 and CentOS 8 Stream in the server. I guess those error come from that.

those errors don’t look good:

deadlock detected
DETAIL:  Process 29839 waits for ShareLock on transaction 42277; blocked by process 29878.
Process 29878 waits for ShareLock on transaction 42195; blocked by process 29839.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1586,5) in relation "rpm_package_pkgId_key"
deadlock detected
DETAIL:  Process 29865 waits for ShareLock on transaction 42277; blocked by process 29878.
Process 29878 waits for ShareLock on transaction 42262; blocked by process 29865.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (2145,3) in relation "rpm_package_pkgId_key"

context in the postgresql log of the content proxy for the first one:

2021-07-07 16:31:24 CEST ERROR:  deadlock detected
2021-07-07 16:31:24 CEST DETAIL:  Process 29839 waits for ShareLock on transaction 42277; blocked by process 29878.
	Process 29878 waits for ShareLock on transaction 42195; blocked by process 29839.
	Process 29839: INSERT INTO "rpm_package" ("content_ptr_id", "name", "epoch", "version", "release", "arch", "pkgId", "checksum_type", "summary", "description", "url", "changelogs", "files", "requires", "provides", "conflicts", "obsoletes", "suggests", "enhances", "recommends", "supplements", "location_base", "location_href", "rpm_buildhost", "rpm_group", "rpm_license", "rpm_packager", "rpm_sourcerpm", "rpm_vendor", "rpm_header_start", "rpm_header_end", "size_archive", "size_installed", "size_package", "time_build", "time_file", "is_modular") VALUES ('a936aa63-3ba4-48e5-8c56-f5bb2e467b72'::uuid, 'tfm-rubygem-http-form_data-doc', '0', '2.1.1', '1.el7', 'noarch', 'a1aeb93ba5471b43ecdc3575a49a43ad7010a90f28a333a13ccb250d57117984', 'sha256', 'Documentation for rubygem-http-form_data', 'Documentation for rubygem-http-form_data.', 'https://github.com/httprb/form_data.rb', '[["Moti Asayag <masayag@redhat.com> 2.1.1-1", 1556107200, "- Add rubygem-http-form_data generated by gem2rpm using the scl template"]]', '[["dir", "/opt/thefore
	Process 29878: INSERT INTO "rpm_package" ("content_ptr_id", "name", "epoch", "version", "release", "arch", "pkgId", "checksum_type", "summary", "description", "url", "changelogs", "files", "requires", "provides", "conflicts", "obsoletes", "suggests", "enhances", "recommends", "supplements", "location_base", "location_href", "rpm_buildhost", "rpm_group", "rpm_license", "rpm_packager", "rpm_sourcerpm", "rpm_vendor", "rpm_header_start", "rpm_header_end", "size_archive", "size_installed", "size_package", "time_build", "time_file", "is_modular") VALUES ('caf772fc-612e-41ff-bdb0-d508afa35e9b'::uuid, 'puppetlabs-stdlib', '0', '5.2.0', '1.el7', 'noarch', '72ebc022f442cd9755d4b920a57aec66c780885e08e1c65cf57e5ea52961d5cd', 'sha256', 'Puppet Labs Standard Library', 'Puppet Labs Standard Library module.', 'https://github.com/puppetlabs/puppetlabs-stdlib', '[["<zhunting@redhat.com> - 5.2.0-1", 1589198400, "- Update Puppetlabs-stdlib to 5.2.0"], ["Andrea Veri <averi@fedoraproject.org> - 4.25.1-1", 1525953600, "- New upstream release."],
2021-07-07 16:31:24 CEST HINT:  See server log for query details.
2021-07-07 16:31:24 CEST CONTEXT:  while inserting index tuple (1586,5) in relation "rpm_package_pkgId_key"
2021-07-07 16:31:29 CEST ERROR:  deadlock detected
2021-07-07 16:31:29 CEST DETAIL:  Process 29865 waits for ShareLock on transaction 42277; blocked by process 29878.
	Process 29878 waits for ShareLock on transaction 42262; blocked by process 29865.
	Process 29865: INSERT INTO "rpm_package" ("content_ptr_id", "name", "epoch", "version", "release", "arch", "pkgId", "checksum_type", "summary", "description", "url", "changelogs", "files", "requires", "provides", "conflicts", "obsoletes", "suggests", "enhances", "recommends", "supplements", "location_base", "location_href", "rpm_buildhost", "rpm_group", "rpm_license", "rpm_packager", "rpm_sourcerpm", "rpm_vendor", "rpm_header_start", "rpm_header_end", "size_archive", "size_installed", "size_package", "time_build", "time_file", "is_modular") VALUES ('0b3a3d45-01b2-46db-b460-9ce2e12d3493'::uuid, 'tfm-rubygem-http-form_data-doc', '0', '2.1.1', '1.el7', 'noarch', 'a1aeb93ba5471b43ecdc3575a49a43ad7010a90f28a333a13ccb250d57117984', 'sha256', 'Documentation for rubygem-http-form_data', 'Documentation for rubygem-http-form_data.', 'https://github.com/httprb/form_data.rb', '[["Moti Asayag <masayag@redhat.com> 2.1.1-1", 1556107200, "- Add rubygem-http-form_data generated by gem2rpm using the scl template"]]', '[["dir", "/opt/thefore
	Process 29878: INSERT INTO "rpm_package" ("content_ptr_id", "name", "epoch", "version", "release", "arch", "pkgId", "checksum_type", "summary", "description", "url", "changelogs", "files", "requires", "provides", "conflicts", "obsoletes", "suggests", "enhances", "recommends", "supplements", "location_base", "location_href", "rpm_buildhost", "rpm_group", "rpm_license", "rpm_packager", "rpm_sourcerpm", "rpm_vendor", "rpm_header_start", "rpm_header_end", "size_archive", "size_installed", "size_package", "time_build", "time_file", "is_modular") VALUES ('adfca120-1641-4999-9d3f-17b94b1a9aed'::uuid, 'tfm-rubygem-azure_mgmt_compute-doc', '0', '0.18.7', '1.el7', 'noarch', '11a4680cfdcc8b89bdc5f7d272a3146d50a3d84a6249288c6a758f1e74ea4e30', 'sha256', 'Documentation for rubygem-azure_mgmt_compute', 'Documentation for rubygem-azure_mgmt_compute.', 'https://aka.ms/azure-sdk-for-ruby', '[["Aditi Puntambekar <apuntamb@redhat.com> 0.18.7-1", 1571745600, "- Add rubygem-azure_mgmt_compute generated by gem2rpm using the scl template"]]', '[[
2021-07-07 16:31:29 CEST HINT:  See server log for query details.
2021-07-07 16:31:29 CEST CONTEXT:  while inserting index tuple (2145,3) in relation "rpm_package_pkgId_key"

The final errors on checksum:

duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 7dcb382f-7da3-40c6-81f4-fbf6e69e3618) already exists.
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 7dcb382f-7da3-40c6-81f4-fbf6e69e3618) already exists.
duplicate key value violates unique constraint "rpm_checksum_path_checksum_distribution_tree_id_fd3fe409_uniq"
DETAIL:  Key (path, checksum, distribution_tree_id)=(images/boot.iso, sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c, 7dcb382f-7da3-40c6-81f4-fbf6e69e3618) already exists.

database:

pulpcore=# select * from rpm_checksum;
               pulp_id                |         pulp_created          |       pulp_last_updated       |           path            |                                checksum                                 |         distribution_tree_id         
--------------------------------------+-------------------------------+-------------------------------+---------------------------+-------------------------------------------------------------------------+--------------------------------------
 46093ffe-b589-4356-bb76-7999c86c414e | 2021-07-07 16:33:48.172646+02 | 2021-07-07 16:33:48.172668+02 | images/boot.iso           | sha256:c79921e24d472144d8f36a0d5f409b12bd016d9d7d022fd703563973ca9c375c | 7dcb382f-7da3-40c6-81f4-fbf6e69e3618
 ef87b8f1-ebee-4d88-bdc3-5099709c84ac | 2021-07-07 16:33:48.172704+02 | 2021-07-07 16:33:48.172713+02 | images/efiboot.img        | sha256:124b70e171cae28449cd1dbee1eebdc7323405c1d0429abb43a34121b2037891 | 7dcb382f-7da3-40c6-81f4-fbf6e69e3618
 7def5f13-58e8-41e5-ba2e-ad7f9bdb4bef | 2021-07-07 16:33:48.172735+02 | 2021-07-07 16:33:48.172743+02 | images/install.img        | sha256:34ad253973ef9023260ddd92c298839b6aef370a6479b8f7c2af6c6c946389c4 | 7dcb382f-7da3-40c6-81f4-fbf6e69e3618
 4992fb8d-751c-4b27-9c64-1bb345d04fc2 | 2021-07-07 16:33:48.172765+02 | 2021-07-07 16:33:48.172774+02 | images/pxeboot/initrd.img | sha256:3c002575bebeca422765197985bb7eaa82fa6bc8703cb10e4c7f4f67247acdc9 | 7dcb382f-7da3-40c6-81f4-fbf6e69e3618
 0bb3157c-8f19-436d-bd53-7e8df43ec94f | 2021-07-07 16:33:48.172795+02 | 2021-07-07 16:33:48.172803+02 | images/pxeboot/vmlinuz    | sha256:6af38d5a53efe5867819b70adb6552c8a7200a6274666177c0ded6b64c1ea04f | 7dcb382f-7da3-40c6-81f4-fbf6e69e3618
(5 rows)
pulpcore=# select * from rpm_distributiontree where content_ptr_id = '7dcb382f-7da3-40c6-81f4-fbf6e69e3618';
            content_ptr_id            | header_version | release_name | release_short | release_version | release_is_layered | base_product_name | base_product_short | base_product_version |  arch  | build_timestamp | instimage |     mainimage      | discnum | totaldiscs 
--------------------------------------+----------------+--------------+---------------+-----------------+--------------------+-------------------+--------------------+----------------------+--------+-----------------+-----------+--------------------+---------+------------
 7dcb382f-7da3-40c6-81f4-fbf6e69e3618 | 1.2            | CentOS Linux | CentOS        | 8               | f                  |                   |                    |                      | x86_64 |      1622579912 |           | images/install.img |         |           
(1 row)

So again, CentOS 8…

the other 3 errors during a later optimized sync of the content proxy doesn’t look so good either:

insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(7db11963-4adf-49df-b276-d66f6f1c24ce) is not present in table "core_repositoryversion".
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(c3fd26f1-7dc3-465a-bf1a-21ca479fb5c1) is not present in table "core_repositoryversion".
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(2505b1f7-cc8a-489d-aaab-911a30910319) is not present in table "core_repositoryversion".

Those uuids don’t exist in core_repositoryversion. It’s a commit error:

2021-07-07 19:04:04 CEST ERROR:  insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
2021-07-07 19:04:04 CEST DETAIL:  Key (version_added_id)=(7db11963-4adf-49df-b276-d66f6f1c24ce) is not present in table "core_repositoryversion".
2021-07-07 19:04:04 CEST STATEMENT:  COMMIT

It’s really weird: the last optimized syncs all worked without errors. After the update to pulpcore 3.14 I even tried a complete sync and it worked. I think I even did a optimized sync after that yesterday.

But today, again, those errors again:

insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(b20330dd-0b45-4415-b9d2-648773d94c24) is not present in table "core_repositoryversion".
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(b7dcd79a-3f73-49b2-8500-b2d91567cd7f) is not present in table "core_repositoryversion".
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(869d1cbe-4033-4ae4-a79c-cc6e9c2de581) is not present in table "core_repositoryversion".

There are some kernel and systemd updates available, so maybe some distribution information or similar in the repositories was updated causing some issue down the way?