O.K. It gets worse. I have removed all environments from the proxy. Sync optimized, complete, and reclaimed. Then ran
# foreman-rake katello:delete_orphaned_content RAILS_ENV=production SMART_PROXY_ID=4
four times. The first time it cleared out a lot. After that, it basically completes immediately.
So there really shouldn’t be anything on the proxy. However, there is still something left:
pulpcore=# select * from core_repository;
pulp_id | pulp_created | pulp_last_updated | name | description | next_version | pulp_type | remote_id | retain_repo_versions | user_hidden
--------------------------------------+-------------------------------+-------------------------------+----------------------------------------------------------------------------+-------------+--------------+-----------+-----------+----------------------+-------------
11e10445-b7a3-4d48-aecf-3ae78a49374b | 2022-07-25 12:59:55.244584+02 | 2022-07-25 14:01:20.511506+02 | AppStream-666405902494e37e80c80752e70918811bab2a89ab2011a50e22fc21926760cd | | 2 | rpm.rpm | | | t
94f2daba-1ae5-4808-99fb-2248c3e8e929 | 2022-07-25 12:59:44.676468+02 | 2022-07-25 13:57:01.049942+02 | AppStream-cafaa7d8a979743d2c39308ba5c31a702ee94aeea4bab81ccb5b4d7a9b668ae7 | | 5 | rpm.rpm | | | t
b706e419-be27-4cdd-b9f8-7d2df7e5baf4 | 2022-07-25 13:25:39.703301+02 | 2022-07-25 14:01:19.850977+02 | AppStream-ecc86edecfd6b3ed311a9eabac6d21c8d313c6f80cf813b703cb1da72613b7a7 | | 2 | rpm.rpm | | | t
(3 rows)
pulpcore=# select count(*) from core_remote;
count
-------
410
(1 row)
pulpcore=# select count(*) from core_artifact ;
count
-------
1308
(1 row)
pulpcore=# select count(*) from core_content;
count
--------
140871
(1 row)
pulpcore=# select repository_ptr_id,last_sync_details from rpm_rpmrepository ;
repository_ptr_id | last_sync_details
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------
94f2daba-1ae5-4808-99fb-2248c3e8e929 | {"url": "https://foreman8.example.com/pulp/content/ORG/Testing/el8-epel8/custom/almalinux8/BaseOS_x86_64_os/", "revision": "1657778086", "sync_policy": "mirror_complete", "download_policy": "on_demand", "repomd_checksum": "7c2ded7e22200c611023891637f53e36b64bf8a112c3d18142c9aeac9
3a9e715", "most_recent_version": 5}
b706e419-be27-4cdd-b9f8-7d2df7e5baf4 | {"url": "https://foreman8.example.com/pulp/content/ORG/Production/el8-epel8/custom/centos8stream/BaseOS_x86_64_os/", "revision": "1657777519", "sync_policy": "mirror_complete", "download_policy": "on_demand", "repomd_checksum": "012d3187dc83ce5da15a88e081c08288e0f9ab90f298cb48783
234552699bc92", "most_recent_version": 1}
11e10445-b7a3-4d48-aecf-3ae78a49374b | {"url": "https://foreman8.example.com/pulp/content/ORG/Production/el8/custom/centos8stream/BaseOS_x86_64_os/", "revision": "1658329738", "sync_policy": "mirror_complete", "download_policy": "on_demand", "repomd_checksum": "cb1ac8fddc809d6d5529749b2e7501239345b692ddef0a881b5d8bb6d
d90fe7b", "most_recent_version": 1}
(3 rows)
So basically, three BaseOS repositories are still there. There seems to be some serious sync issue with the BaseOS repository. On the main server I have Alma 8 and CentOS Stream 8, in testing and production environments, in content view el8 and el8-epel8. So technically there are a total of 8 remote repositories to sync from the main server. Of those 8, 3 have been left behind.
The 94f2daba-1ae5-4808-99fb-2248c3e8e929 is in the first error.
How do I get this fixed or cleared out? Otherwise, I would simply reinstall the content proxy as it’s yet in production anyway…