Sync errors on all syncs including the initial sync between new katello server and content proxy

Thanks. For the fun of it I have applied Fix a bug preventing orphan cleanup for some subrepos · pulp/pulp_rpm@d41c097 · GitHub on the content proxy and ran katello:delete_orphaned_content again, but the repositories remain there. I didn’t find a patch regarding user_hidden=t. Is there another patch I have to apply.

I really would like to test that it’s completely removing everything if I remove all environments from the content proxy.

How would I delete those 3 repositories? Simply in the database or is there a cli command for that?

Later I can reinstall the content proxy and try the initial sync again. Currently, it looks to me as if it is related to promoting the same CV into multiple synced environments at the same time. I had occasional sync errors on my production server (katello 4.4 and before) which I mostly avoided by putting a five minute delay between the first promotion and the second. That would match what I saw on the new installation.

However, I also checked the errors on my production server and they are always related to the AlmaLinux 8 or CentOS Stream 8 BaseOS repository.

For one error on the production server I have also just noticed something odd:

duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL:  Key (repository_id, number)=(ac46caa8-0b8b-4da6-bcb0-e2e40ec77e19, 12) already exists.
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL:  Key (repository_id, number)=(ac46caa8-0b8b-4da6-bcb0-e2e40ec77e19, 12) already exists.

Database on production content proxy:

pulpcore=# select * from core_repository where pulp_id = 'ac46caa8-0b8b-4da6-bcb0-e2e40ec77e19';
               pulp_id                |         pulp_created          |       pulp_last_updated       |                                    name                                    | description | next_version | pulp_type | remote_id | retain_repo_versions | use
r_hidden 
--------------------------------------+-------------------------------+-------------------------------+----------------------------------------------------------------------------+-------------+--------------+-----------+-----------+----------------------+----
---------
 ac46caa8-0b8b-4da6-bcb0-e2e40ec77e19 | 2022-05-13 03:55:29.965914+02 | 2022-07-26 04:14:37.209507+02 | AppStream-cafaa7d8a979743d2c39308ba5c31a702ee94aeea4bab81ccb5b4d7a9b668ae7 |             |           15 | rpm.rpm   |           |                      | t
(1 row)

pulpcore=# select * from rpm_rpmrepository where repository_ptr_id = 'ac46caa8-0b8b-4da6-bcb0-e2e40ec77e19';
          repository_ptr_id           | metadata_signing_service_id |                                                     original_checksum_types                                                     | retain_package_versions | autopublish | gpgcheck | metadata_
checksum_type | package_checksum_type | repo_gpgcheck | sqlite_metadata |                                                                                                                                                    last_sync_details                      
                                                                                                                               
--------------------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------+-------------------------+-------------+----------+----------
--------------+-----------------------+---------------+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------
 ac46caa8-0b8b-4da6-bcb0-e2e40ec77e19 |                             | {"group": "sha256", "other": "sha256", "modules": "sha256", "primary": "sha256", "filelists": "sha256", "updateinfo": "sha256"} |                       0 | f           |        0 |          
              |                       |             0 | f               | {"url": "https://foreman.example.com/pulp/content/ORG/Testing/el8-epel8/custom/almalinux8/BaseOS_x86_64_os/", "revision": "1658774516", "sync_policy": "mirror_complete", "download_policy": 
"on_demand", "repomd_checksum": "80e8bc521df91d3bf99d41ca3172f93ef72fd8104083bfe6090d079a1f56f34f", "most_recent_version": 14}
(1 row)

Why is the BaseOS repository named AppStream-cafaa7d8a979743d2c39308ba5c31a702ee94aeea4bab81ccb5b4d7a9b668ae7?