After I have removed all environments from my content proxy and it eventually really was completely empty, I have just tried again for the fun of it as I have just added CentOS Stream 9 repositories.
I have got 11 errors:
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(cf38fff6-0d34-461d-ac6a-e6f4b4916013) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 1076
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(98012f5e-ceb2-49b7-b116-b142246c115c) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 1449
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL: Key (repository_id, number)=(b34d636c-73d1-4fbc-bdb1-5d52699ca1fc, 1) already exists.
Could not lookup a publication_href for repo 480
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL: Key (repository_id, number)=(b34d636c-73d1-4fbc-bdb1-5d52699ca1fc, 1) already exists.
Could not lookup a publication_href for repo 1490
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL: Key (repository_id, number)=(b34d636c-73d1-4fbc-bdb1-5d52699ca1fc, 1) already exists.
Could not lookup a publication_href for repo 1629
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(bdf1fdf3-51a0-4a25-9c66-9fc7edab1be9) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 429
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(6050a3f9-7f37-405c-b9ff-16d8a971ae2b) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 439
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(b9b59f21-2c96-46b1-90c3-b4e01fe75d52) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 1794
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(3aaf24b8-0283-4463-90e2-78d171461750) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 490
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL: Key (repository_id, number)=(a3d96d2e-b2bd-4fb2-a079-ae5dbaa44ba4, 1) already exists.
Could not lookup a publication_href for repo 1937
insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(a7301b58-be8d-406f-8dfe-22b43bbba781) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 1903
which all match BaseOS repositories:
foreman=# select id,relative_path from katello_repositories where id in ( 1076,1449,480,1490,1629,429,430,1794,490,1937,1903) order by relative_path;
id | relative_path
------+-----------------------------------------------------------------
1490 | ORG/Production/alma8/custom/almalinux8/BaseOS_x86_64_os
1629 | ORG/Production/alma8-epel8/custom/almalinux8/BaseOS_x86_64_os
1794 | ORG/Production/cs8/custom/centos8stream/BaseOS_x86_64_os
1937 | ORG/Production/cs8-epel8/custom/centos8stream/BaseOS_x86_64_os
480 | ORG/Production/el8/custom/almalinux8/BaseOS_x86_64_os
490 | ORG/Production/el8/custom/centos8stream/BaseOS_x86_64_os
1449 | ORG/Testing/alma8/custom/almalinux8/BaseOS_x86_64_os
1903 | ORG/Testing/cs8-epel8/custom/centos8stream/BaseOS_x86_64_os
430 | ORG/Testing/el8/custom/almalinux8/AppStream_x86_64_os
429 | ORG/Testing/el8/custom/almalinux8/BaseOS_x86_64_os
1076 | ORG/Testing/el8-epel8/custom/almalinux8/BaseOS_x86_64_os
(11 rows)
Available on the content proxy are the following BaseOS repositories, taken from the output of curl -s 'http://foreman8-content.dkrz.de/pulp/content/' | grep 'ORG.*BaseOS'
ORG/Production/cs9-epel9/custom/centos9/BaseOS_x86_64_os/
ORG/Testing/alma8-epel8/custom/almalinux8/BaseOS_x86_64_os/
ORG/Testing/cs8/custom/centos8stream/BaseOS_x86_64_os/
ORG/Testing/cs9-epel9/custom/centos9/BaseOS_x86_64_os/
ORG/Testing/el8-epel8/custom/centos8stream/BaseOS_x86_64_os/
which are exactly those 5 repositories for which there was no error. Notably, both CentOS Stream 9 BaseOS repos got through. I haven’t tested more, yet, so it could be just coincidence.
The two uuids b34d636c-73d1-4fbc-bdb1-5d52699ca1fc and a3d96d2e-b2bd-4fb2-a079-ae5dbaa44ba4 for which there was a duplicate key value
pulpcore=# select * from core_repository where pulp_id in ('b34d636c-73d1-4fbc-bdb1-5d52699ca1fc','a3d96d2e-b2bd-4fb2-a079-ae5dbaa44ba4');
pulp_id | pulp_created | pulp_last_updated |
name | description | next_version | pulp_type | remote_id | retain_repo_versions | user_hi
dden
--------------------------------------+-------------------------------+-------------------------------+-----------------------------
-----------------------------------------------+-------------+--------------+-----------+-----------+----------------------+--------
-----
b34d636c-73d1-4fbc-bdb1-5d52699ca1fc | 2022-08-16 08:20:46.476563+02 | 2022-08-16 08:23:47.698258+02 | AppStream-cafaa7d8a979743d2c
39308ba5c31a702ee94aeea4bab81ccb5b4d7a9b668ae7 | | 2 | rpm.rpm | | | t
a3d96d2e-b2bd-4fb2-a079-ae5dbaa44ba4 | 2022-08-16 08:25:18.121076+02 | 2022-08-16 08:33:44.662956+02 | AppStream-0312709cc54e5821c0
7aca3160d7da33860d901a8348c493e7bb2949bbc60b97 | | 2 | rpm.rpm | | | t
(2 rows)
both are named AppStream-… as you can see, even though they point to BaseOS repositories:
pulpcore=# select last_sync_details from rpm_rpmrepository where repository_ptr_id in ('b34d636c-73d1-4fbc-bdb1-5d52699ca1fc','a3d96d2e-b2bd-4fb2-a079-ae5dbaa44ba4');
last_sync_details
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------
{"url": "https://foreman8.example.com/pulp/content/ORG/Testing/alma8-epel8/custom/almalinux8/BaseOS_x86_64_os/", "revision": "16597106
60", "sync_policy": "mirror_complete", "download_policy": "on_demand", "repomd_checksum": "2dbc15a2c91e47985e09af92f66f6bbc342942668
c91b56e4dfb8784ade14112", "most_recent_version": 1}
{"url": "https://foreman8.example.com/pulp/content/ORG/Testing/cs8/custom/centos8stream/BaseOS_x86_64_os/", "revision": "1659711971",
"sync_policy": "mirror_complete", "download_policy": "on_demand", "repomd_checksum": "d2f72b0905bb16fedfc0c70187306c868ad7d097e469bd
d1e0bdb9827e4244ff", "most_recent_version": 1}
(2 rows)
So it’s always BaseOS repositories with those issues and they are linked in some way with AppStream…