Finally finished content migration but then newly added repos are not available to clients

To be able to complete the content migration, I had to remove some of the repos (e.g. CentOS 8 AppStream, …) because of issues with the .treeinfo file. After I’ve removed them, the migration was successful.
Then I added the previously removed repos again but they are not accessible on my clients. I can see them when running a “dnf repolist”:
# dnf repolist
Updating Subscription Management repositories.
repo id repo name
MyCompany_CentOS_8_AppStream_x86_64 CentOS 8 AppStream x86_64
MyCompany_CentOS_8_BaseOS_x86_64 CentOS 8 BaseOS x86_64
MyCompany_CentOS_8_Extras_x86_64 CentOS 8 Extras x86_64
MyCompany_CentOS_8_PowerTools_x86_64 CentOS 8 PowerTools x86_64
MyCompany_EL8_-_lldpd_lldpd_x86_64 EL8 lldpd x86_64
MyCompany_EPEL_8_Everything_x86_64 EPEL 8 Everything x86_64
MyCompany_EPEL_8_Modular_x86_64 EPEL 8 Modular x86_64
MyCompany_Elastic_6_Elastic_7_x86_64 Elastic 7
MyCompany_Foreman_Client_EL8_Foreman_Client_2_0_x86_64 Foreman Client 2.0 x86_64
MyCompany_Foreman_Client_EL8_Foreman_Client_2_3_x86_64 Foreman Client 2.3 x86_64
MyCompany_MyCompany_EL8_MyCompany_Local MyCompany Local
MyCompany_Microsoft_EL8_Productive_x86_64 Microsoft EL8 Productive x86_64
MyCompany_Microsoft_EL8_RHEL_8_Productive_x86_64 Microsoft RHEL 8 Productive x86_64
MyCompany_Puppet_v6_EL8_Puppet_v6_x86_64 EL8 Puppet v6 x86_64
foreman-client Foreman client 2.3

But when trying to update, the new repos are not accessible:
# yum check-update
Updating Subscription Management repositories.
CentOS 8 Extras x86_64 961 B/s | 224 B 00:00
Errors during downloading metadata for repository ‘MyCompany_CentOS_8_Extras_x86_64’:
- Status code: 404 for https://stg-katello.mylan.lan/pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml (IP:
Error: Failed to download metadata for repo ‘MyCompany_CentOS_8_Extras_x86_64’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

foreman-ssl_access_ssl.log shows: - - [23/Apr/2021:09:16:46 +0200] “GET /rhsm/consumers/5c48a24c-ea50-48c2-97f6-7aace879e016/certificates/serials HTTP/1.1” 500 77 “-” “RHSM/1.0 (cmd=yum)” - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)” - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)” - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)” - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”

when I try the update with all newly added repos disabled, it runs successfully and in foreman-ssl_access_ssl.log I see: - - [23/Apr/2021:09:17:55 +0200] “GET /rhsm/consumers/5c48a24c-ea50-48c2-97f6-7aace879e016/certificates/serials HTTP/1.1” 500 77 “-” “RHSM/1.0 (cmd=yum)” - - [23/Apr/2021:09:17:56 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/Elastic_6/Elastic_7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2160 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)” - - [23/Apr/2021:09:17:56 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/BaseOS_x86_64/repodata/repomd.xml HTTP/1.1” 200 2164 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”

I already tried to re-register the clients but the issue is still the same, migrated repos are accessible, newly added are not.

Expected outcome:
all repos accessible to clients

Foreman and Proxy versions:
Foreman: 2.3.3
Katello: 3.18.2


This text will be hidden

Edit: Redacted to remove some personal info

1 Like

That could be the same issue I see (topic [ContentMigration] switchover fails at katello:pulp3_post_migration_check at the end)

On my katello server I am not able get any published/promoted content view version into the available repos. They are just missing completely, having only those packages available which have been in the last version before the migration.

IMHO this is an issue of understanding (or documentation) and not a technical issue :wink:

For me it was not clear that, after the content switchover, I immediately have to upgrade to Katello 4.

I’ve upgraded my Katello now to version 4 and everything is now working again.

Maybe someone can add this to the documentation?

O.K. That is good to know. But from the documentation as well as discussions in this community, I think the pulp3 migration should be possible at any time. The migration was first introduced with Katello 3.15. Thus, it’s unlikely that it depends on Katello 4.

I guess, either Katello 4 has an updated pulpcore version which fixes this issue or maybe the final cleanup of all the pulp2 stuff in the katello 4 installer solves the problems.

Either way, I already though about updating to katello 4 regardless of the existing issue. It’s good to know it may solve the problem…