Greetings to all,
Lately, we are getting 404’s when trying to install packages on a client system using Yum. The package exists in Katello but is not in the pulp directory.
I get a 404 on this RPM.
However, it does exist in katello itself, has worked with this very same Content View in the past and works with other clients registered through a Smart Proxy. The clients currently having an issue are registered to the main katello server itself.
That a package that is in a Content View can be installed and that we don’t get 404’s when visiting the pulp directory where it is hosted out of.
Foreman and Proxy versions:
Foreman 3.1.3 and Katello 4.3.1
Foreman and Proxy plugin versions:
Distribution and version:
Any ideas on what may be causing this? It is rather perplexing. Thank you
Here is the package in question, we can see it is indeed in Katello and has been installed on thousands of hosts. Currently this package can be installed if the client is registered thru a Smart Proxy (using the same exact CV in fact) but if a client is registered to the main katello server itself, we get a 404.
Here it shows the package as available from the main server itself (visiting the pulp URL in a browser) but if I click it, I get a 404. Same error as when a client tries to install it via YUM.
Hopefully someone can shine some light on this
Hi @eledor ,
One note – Katello 4.3 is quite old at this point, we’re currently releasing Katello 4.11. We consider the newest two releases to be the ones “in support”.
Anyway, what content mirroring type and what download policy are you using for your repository? Is it possible that you used “on-demand” but the RPMs no longer exist in the upstream repository?
If your smart proxy has the RPMs I’m guessing they were on disk at one point, so maybe you used
reclaim space to delete the old RPMs?
If that’s not it, I would recommend trying to “republish” the content view repositories. I can’t remember if Katello 4.3 supports it, but the hammer command would be
hammer content-view version republish-repositories ...
If hammer doesn’t support it it should be in the API docs.
Thanks for replying to me.
For the CentOS 7 base repo, where we are pulling from the, the Download Policy is Immediate the Mirroring Policy is Additive.
Also, I checked the upstream repo where we are pulling from and that exact package is still up there:
I have tried republishing the existing Content View along with creating a brand new one with the same repos added to it and I am getting the same thing.
Any other suggestions would be great.
hi again @iballou
b00m! That was it:
hammer content-view version republish-repositories --id 3081 --force true
(i had previously just tried republishing the CV from the GUI but this hammer command did the trick).
any ideas on what may have caused this?
also, is there a way i can run this safely on the server for all CV’s? Right now I am doing it per CV version.
per CV, regardless of version, would be ideal.
Obviously I can’t be sure, but the fact that a “republish repository metadata” action fixed it, along with the fact that you could see the package being served, but got a 404 when trying to download it, suggests that the Pulp publication included an artifact reference to a missing file. Perhaps something was cleaned up in error. The hammer command that fixed it causes Katello to ask Pulp to re-create the Pulp publication that is served, which clearly fixed the issue.
Glad it worked! We only have it available per content view version. It is odd that it happened across much of your system. I can only guess that Pulp erroneously cleaned up some artifacts during orphan cleanup.
Hi @quba42 and @iballou
Unfortunately, this issue continues to happen and perplex us. Any more insight you may be able to provide would be most welcome.
What I can confirm is the following:
Publishing a new Content View will trigger this every time. This used to not be the case.
We will get the 404 error of a package that shows as available from the URL.
Running republish-repositories with hammer on the Content Views fixes this until a new CV is published. Therefore, I don’t think it is a orphan cleanup triggering this?
The repmod.xml file after the CV publish is old and a yum install produces these kinds of errors:
[19:33:32 root@memcache03 ~]# yum install vim
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager, tsflags, versionlock
Loading mirror speeds from cached hostfile
Cloud_SRE_CentOS_7_CentOS_7_base_x86_64 | 2.3 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_base_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:45 2023
Downloaded: Fri Aug 5 08:47:39 2022
Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64 | 2.0 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:33 2023
Downloaded: Fri Jul 7 06:31:31 2023
Running a yum clean all on the client will make these errors go away but I think we are just using the old repmod.xml at that point. Republishing the repos will update this repmod.xml and then the clients are happy.
- Our current mirroring policy is “Additive”. Could this be part of the issue? Do you think it wise to change it to Content Only and can this be safely done?