Lately, we are getting 404’s when trying to install packages on a client system using Yum. The package exists in Katello but is not in the pulp directory.
However, it does exist in katello itself, has worked with this very same Content View in the past and works with other clients registered through a Smart Proxy. The clients currently having an issue are registered to the main katello server itself.
Expected outcome:
That a package that is in a Content View can be installed and that we don’t get 404’s when visiting the pulp directory where it is hosted out of.
Foreman and Proxy versions:
Foreman 3.1.3 and Katello 4.3.1
Foreman and Proxy plugin versions:
Distribution and version:
Rocky 8
Any ideas on what may be causing this? It is rather perplexing. Thank you
Here is the package in question, we can see it is indeed in Katello and has been installed on thousands of hosts. Currently this package can be installed if the client is registered thru a Smart Proxy (using the same exact CV in fact) but if a client is registered to the main katello server itself, we get a 404.
Here it shows the package as available from the main server itself (visiting the pulp URL in a browser) but if I click it, I get a 404. Same error as when a client tries to install it via YUM.
One note – Katello 4.3 is quite old at this point, we’re currently releasing Katello 4.11. We consider the newest two releases to be the ones “in support”.
Anyway, what content mirroring type and what download policy are you using for your repository? Is it possible that you used “on-demand” but the RPMs no longer exist in the upstream repository?
If your smart proxy has the RPMs I’m guessing they were on disk at one point, so maybe you used reclaim space to delete the old RPMs?
If that’s not it, I would recommend trying to “republish” the content view repositories. I can’t remember if Katello 4.3 supports it, but the hammer command would be hammer content-view version republish-repositories ...
If hammer doesn’t support it it should be in the API docs.
I have tried republishing the existing Content View along with creating a brand new one with the same repos added to it and I am getting the same thing.
Obviously I can’t be sure, but the fact that a “republish repository metadata” action fixed it, along with the fact that you could see the package being served, but got a 404 when trying to download it, suggests that the Pulp publication included an artifact reference to a missing file. Perhaps something was cleaned up in error. The hammer command that fixed it causes Katello to ask Pulp to re-create the Pulp publication that is served, which clearly fixed the issue.
Glad it worked! We only have it available per content view version. It is odd that it happened across much of your system. I can only guess that Pulp erroneously cleaned up some artifacts during orphan cleanup.
Unfortunately, this issue continues to happen and perplex us. Any more insight you may be able to provide would be most welcome.
What I can confirm is the following:
Publishing a new Content View will trigger this every time. This used to not be the case.
We will get the 404 error of a package that shows as available from the URL.
Running republish-repositories with hammer on the Content Views fixes this until a new CV is published. Therefore, I don’t think it is a orphan cleanup triggering this?
The repmod.xml file after the CV publish is old and a yum install produces these kinds of errors:
[19:33:32 root@memcache03 ~]# yum install vim
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager, tsflags, versionlock
Loading mirror speeds from cached hostfile
Cloud_SRE_CentOS_7_CentOS_7_base_x86_64 | 2.3 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_base_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:45 2023
Downloaded: Fri Aug 5 08:47:39 2022
Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64 | 2.0 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:33 2023
Downloaded: Fri Jul 7 06:31:31 2023
Running a yum clean all on the client will make these errors go away but I think we are just using the old repmod.xml at that point. Republishing the repos will update this repmod.xml and then the clients are happy.
Our current mirroring policy is “Additive”. Could this be part of the issue? Do you think it wise to change it to Content Only and can this be safely done?
This is only happening for our CentOS 7 Repos. Rocky 8 does not exhibit this behavior so it is not system wide.
Republishing a CV and then trying to install a package through a Smart Proxy (not directly registered to the main katello server) we don’t have this problem. The problem only manifests itself for CentOS 7 Content Views and hosts directly registered to the main server.
This does not make any sense to me. “Running republish-repositories with hammer on the Content Views” just re-runs one of the steps that is performed when Publishing a new Content View (at least on a current version). It is inconceivable to me that this should result in different results, unless the very old version you are using contains bugs that have since been fixed.
The truth is that you are unlikely to find anyone who is going to help you with this for Foreman 3.1/pulpcore 3.16. My sense is that it would take a Programmer well versed in both Katello and Pulp (with access to the system) several hours to have a good shot at debugging this.
Yeah like @quba42 said we’ve had improvements in our self-healing since Katello 4.3. It would be a lot easier to help out if you’re able to upgrade to at least Katello 4.9. The code your own won’t be receiving any more fixes, but if you upgrade to newer code and that still breaks, it makes it a lot easier to give priority.
Something must be corrupt in your DB that’s confusing our CV publishing code I’m assuming. To tell for sure we’d need to poke around the database.
In the foreman console (foreman-rake console) you can do things like:
# List repositories linked to published repositories
repos = ::Katello::ContentViewVersion.find(<id>).repositories.archived
# Then, for example:
# Show pulp version_href
repos.first.version_href
# Show pulp publication_href
repos.first.publication_href
# show pulp distributions
repos.first.distribution_references
