Problem:
For migration from EL7 to EL8 I have installed and configured a new Katello server with 4.5.0 and added a content proxy also with 4.5.0. Repositories, environments, etc. have been set up with the ansible modules. Now I have run the very first syncs to get my testing and production environments synced to the content proxy.
However, I always get sync errors. I have tried multiple times, either optimized sync or complete sync but each time I get one or more errors, except for a single optimized sync in between.
- Sync (optimized?) started by publish and promote:
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL: Key (repository_id, number)=(94f2daba-1ae5-4808-99fb-2248c3e8e929, 1) already exists.
Could not lookup a publication_href for repo 429
- Sync (optimized?) started at almost the same time as no. 1 by publish and promote:
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(b21d5610-806a-4f0b-abd5-467741881dfd) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 490
- Manual complete sync
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_removed_id_4d75bc32_fk_core_repo"
DETAIL: Key (version_removed_id)=(28b7e2a4-54ad-4357-aa5f-bd03c58b1ff6) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 1119insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(c9ed5dab-ece0-48be-9d4e-f5fdb6c2abb0) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 1090
- Yet, another manual complete sync after no. 3 finished with an error:
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL: Key (repository_id, number)=(94f2daba-1ae5-4808-99fb-2248c3e8e929, 4) already exists.
Could not lookup a publication_href for repo 1119
- Manual, optimized sync with no errors.
- Manual complete sync:
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(25e11040-2be6-4f0a-8141-8e870d7fa430) is not present in table "core_repositoryversion".
- Manual optimized sync:
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL: Key (repository_id, number)=(94f2daba-1ae5-4808-99fb-2248c3e8e929, 5) already exists.
Now how do I get this fixed?
Expected outcome:
No sync errors.
Foreman and Proxy versions:
Foreman 3.3.0, Katello 4.5.0 up-to-date
Distribution and version:
AlmaLinux 8.6
Other relevant data:
All problems seem to be related with AlmaLinux 8 and CentOS Stream 8 BaseOS:
I dug up the ids from the databases on the main server and the content proxy:
94f2daba-1ae5-4808-99fb-2248c3e8e929
pulpcore=# select repository_ptr_id,last_sync_details from rpm_rpmrepository where repository_ptr_id = '94f2daba-1ae5-4808-99fb-2248c3e8e929';
repository_ptr_id | last_sync_details
--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------
94f2daba-1ae5-4808-99fb-2248c3e8e929 | {"url": "https://foreman8.example.com/pulp/content/ORG/Testing/el8-epel8/custom/almalinux8/BaseOS_x86_64_os/"...
which matches repo 429 on the main server:
foreman=# select id,relative_path,publication_href from katello_repositories where id in (429);
id | relative_path | publication_href
-----+-----------------------------------------------------+-------------------------------------------------------------------------
429 | ORG/Testing/el8/custom/almalinux8/BaseOS_x86_64_os | /pulp/api/v3/publications/rpm/rpm/942ffc44-dd34-4403-8b7d-84662d8cb6e6/
b21d5610-806a-4f0b-abd5-467741881dfd doesn’t exist (anymore?) in the database.
but repo 490 is
foreman=# select id,relative_path,publication_href from katello_repositories where id in (490);
id | relative_path | publication_href
-----+-----------------------------------------------------------+-------------------------------------------------------------------------
490 | ORG/Production/el8/custom/centos8stream/BaseOS_x86_64_os | /pulp/api/v3/publications/rpm/rpm/e242c2d0-ab31-4a0f-ac5b-c79257ab221c/
Neither are 28b7e2a4-54ad-4357-aa5f-bd03c58b1ff6, c9ed5dab-ece0-48be-9d4e-f5fdb6c2abb0, 25e11040-2be6-4f0a-8141-8e870d7fa430
pulpcore=# select * from core_repositoryversion where pulp_id in ( '28b7e2a4-54ad-4357-aa5f-bd03c58b1ff6', 'c9ed5dab-ece0-48be-9d4e-f5fdb6c2abb0', '25e11040-2be6-4f0a-8141-8e870d7fa430');
pulp_id | pulp_created | pulp_last_updated | number | complete | base_version_id | repository_id
---------+--------------+-------------------+--------+----------+-----------------+---------------
(0 rows)
But repo 1119 and repo 1090 are the other two baseos repositories:
foreman=# select id,relative_path,publication_href from katello_repositories where id in (1119,1090);
id | relative_path | publication_href
------+--------------------------------------------------------------+-------------------------------------------------------------------------
1090 | ORG/Testing/el8-epel8/custom/centos8stream/BaseOS_x86_64_os | /pulp/api/v3/publications/rpm/rpm/4a5ee05b-7966-4c48-8eb1-74545536c754/
1119 | ORG/Production/el8-epel8/custom/almalinux8/BaseOS_x86_64_os | /pulp/api/v3/publications/rpm/rpm/b8142cdb-ff00-4c93-a6bd-a0eaabac6214/
(2 rows)
So it seems to evolve around repo id 94f2daba-1ae5-4808-99fb-2248c3e8e929 affecting all other BaseOS versions. I guess the first two parallels syncs initiated by cv promotions caused some inconsistency in the database and all following syncs cannot straighten it out.