Lately, we are getting 404’s when trying to install packages on a client system using Yum. The package exists in Katello but is not in the pulp directory.
However, it does exist in katello itself, has worked with this very same Content View in the past and works with other clients registered through a Smart Proxy. The clients currently having an issue are registered to the main katello server itself.
Expected outcome:
That a package that is in a Content View can be installed and that we don’t get 404’s when visiting the pulp directory where it is hosted out of.
Foreman and Proxy versions:
Foreman 3.1.3 and Katello 4.3.1
Foreman and Proxy plugin versions:
Distribution and version:
Rocky 8
Any ideas on what may be causing this? It is rather perplexing. Thank you
Here is the package in question, we can see it is indeed in Katello and has been installed on thousands of hosts. Currently this package can be installed if the client is registered thru a Smart Proxy (using the same exact CV in fact) but if a client is registered to the main katello server itself, we get a 404.
Here it shows the package as available from the main server itself (visiting the pulp URL in a browser) but if I click it, I get a 404. Same error as when a client tries to install it via YUM.
One note – Katello 4.3 is quite old at this point, we’re currently releasing Katello 4.11. We consider the newest two releases to be the ones “in support”.
Anyway, what content mirroring type and what download policy are you using for your repository? Is it possible that you used “on-demand” but the RPMs no longer exist in the upstream repository?
If your smart proxy has the RPMs I’m guessing they were on disk at one point, so maybe you used reclaim space to delete the old RPMs?
If that’s not it, I would recommend trying to “republish” the content view repositories. I can’t remember if Katello 4.3 supports it, but the hammer command would be hammer content-view version republish-repositories ...
If hammer doesn’t support it it should be in the API docs.
I have tried republishing the existing Content View along with creating a brand new one with the same repos added to it and I am getting the same thing.
Obviously I can’t be sure, but the fact that a “republish repository metadata” action fixed it, along with the fact that you could see the package being served, but got a 404 when trying to download it, suggests that the Pulp publication included an artifact reference to a missing file. Perhaps something was cleaned up in error. The hammer command that fixed it causes Katello to ask Pulp to re-create the Pulp publication that is served, which clearly fixed the issue.
Glad it worked! We only have it available per content view version. It is odd that it happened across much of your system. I can only guess that Pulp erroneously cleaned up some artifacts during orphan cleanup.
Unfortunately, this issue continues to happen and perplex us. Any more insight you may be able to provide would be most welcome.
What I can confirm is the following:
Publishing a new Content View will trigger this every time. This used to not be the case.
We will get the 404 error of a package that shows as available from the URL.
Running republish-repositories with hammer on the Content Views fixes this until a new CV is published. Therefore, I don’t think it is a orphan cleanup triggering this?
The repmod.xml file after the CV publish is old and a yum install produces these kinds of errors:
[19:33:32 root@memcache03 ~]# yum install vim
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager, tsflags, versionlock
Loading mirror speeds from cached hostfile
Cloud_SRE_CentOS_7_CentOS_7_base_x86_64 | 2.3 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_base_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:45 2023
Downloaded: Fri Aug 5 08:47:39 2022
Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64 | 2.0 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:33 2023
Downloaded: Fri Jul 7 06:31:31 2023
Running a yum clean all on the client will make these errors go away but I think we are just using the old repmod.xml at that point. Republishing the repos will update this repmod.xml and then the clients are happy.
Our current mirroring policy is “Additive”. Could this be part of the issue? Do you think it wise to change it to Content Only and can this be safely done?
This is only happening for our CentOS 7 Repos. Rocky 8 does not exhibit this behavior so it is not system wide.
Republishing a CV and then trying to install a package through a Smart Proxy (not directly registered to the main katello server) we don’t have this problem. The problem only manifests itself for CentOS 7 Content Views and hosts directly registered to the main server.
