Finally finished content migration but then newly added repos are not available to clients

Support
1

Problem:
To be able to complete the content migration, I had to remove some of the repos (e.g. CentOS 8 AppStream, …) because of issues with the .treeinfo file. After I’ve removed them, the migration was successful.
Then I added the previously removed repos again but they are not accessible on my clients. I can see them when running a “dnf repolist”:
# dnf repolist
Updating Subscription Management repositories.
repo id repo name
MyCompany_CentOS_8_AppStream_x86_64 CentOS 8 AppStream x86_64
MyCompany_CentOS_8_BaseOS_x86_64 CentOS 8 BaseOS x86_64
MyCompany_CentOS_8_Extras_x86_64 CentOS 8 Extras x86_64
MyCompany_CentOS_8_PowerTools_x86_64 CentOS 8 PowerTools x86_64
MyCompany_EL8_-_lldpd_lldpd_x86_64 EL8 lldpd x86_64
MyCompany_EPEL_8_Everything_x86_64 EPEL 8 Everything x86_64
MyCompany_EPEL_8_Modular_x86_64 EPEL 8 Modular x86_64
MyCompany_Elastic_6_Elastic_7_x86_64 Elastic 7
MyCompany_Foreman_Client_EL8_Foreman_Client_2_0_x86_64 Foreman Client 2.0 x86_64
MyCompany_Foreman_Client_EL8_Foreman_Client_2_3_x86_64 Foreman Client 2.3 x86_64
MyCompany_MyCompany_EL8_MyCompany_Local MyCompany Local
MyCompany_Microsoft_EL8_Productive_x86_64 Microsoft EL8 Productive x86_64
MyCompany_Microsoft_EL8_RHEL_8_Productive_x86_64 Microsoft RHEL 8 Productive x86_64
MyCompany_Puppet_v6_EL8_Puppet_v6_x86_64 EL8 Puppet v6 x86_64
foreman-client Foreman client 2.3

But when trying to update, the new repos are not accessible:
# yum check-update
Updating Subscription Management repositories.
CentOS 8 Extras x86_64 961 B/s | 224 B 00:00
Errors during downloading metadata for repository ‘MyCompany_CentOS_8_Extras_x86_64’:
- Status code: 404 for https://stg-katello.mylan.lan/pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml (IP: 10.164.250.113)
Error: Failed to download metadata for repo ‘MyCompany_CentOS_8_Extras_x86_64’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

foreman-ssl_access_ssl.log shows:
10.164.17.145 - - [23/Apr/2021:09:16:46 +0200] “GET /rhsm/consumers/5c48a24c-ea50-48c2-97f6-7aace879e016/certificates/serials HTTP/1.1” 500 77 “-” “RHSM/1.0 (cmd=yum)”
10.164.17.145 - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
10.164.17.145 - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
10.164.17.145 - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
10.164.17.145 - - [23/Apr/2021:09:16:47 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/Extras_x86_64/repodata/repomd.xml HTTP/1.1” 404 224 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”

when I try the update with all newly added repos disabled, it runs successfully and in foreman-ssl_access_ssl.log I see:
10.164.17.145 - - [23/Apr/2021:09:17:55 +0200] “GET /rhsm/consumers/5c48a24c-ea50-48c2-97f6-7aace879e016/certificates/serials HTTP/1.1” 500 77 “-” “RHSM/1.0 (cmd=yum)”
10.164.17.145 - - [23/Apr/2021:09:17:56 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/Elastic_6/Elastic_7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2160 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
10.164.17.145 - - [23/Apr/2021:09:17:56 +0200] “GET /pulp/repos/MyCompany/Library/Enterprise_Linux_8_Rolling_Updates/custom/CentOS_8/BaseOS_x86_64/repodata/repomd.xml HTTP/1.1” 200 2164 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
…

I already tried to re-register the clients but the issue is still the same, migrated repos are accessible, newly added are not.

Expected outcome:
all repos accessible to clients

Foreman and Proxy versions:
Foreman: 2.3.3
Katello: 3.18.2

Summary

This text will be hidden

Edit: Redacted to remove some personal info

1 Like
2

That could be the same issue I see (topic [ContentMigration] switchover fails at katello:pulp3_post_migration_check at the end)

On my katello server I am not able get any published/promoted content view version into the available repos. They are just missing completely, having only those packages available which have been in the last version before the migration.

3

IMHO this is an issue of understanding (or documentation) and not a technical issue :wink:

For me it was not clear that, after the content switchover, I immediately have to upgrade to Katello 4.

I’ve upgraded my Katello now to version 4 and everything is now working again.

Maybe someone can add this to the documentation?

4

O.K. That is good to know. But from the documentation as well as discussions in this community, I think the pulp3 migration should be possible at any time. The migration was first introduced with Katello 3.15. Thus, it’s unlikely that it depends on Katello 4.

I guess, either Katello 4 has an updated pulpcore version which fixes this issue or maybe the final cleanup of all the pulp2 stuff in the katello 4 installer solves the problems.

Either way, I already though about updating to katello 4 regardless of the existing issue. It’s good to know it may solve the problem…