RPM's have started to go missing from Content Views

Greetings to all,


Lately, we are getting 404’s when trying to install packages on a client system using Yum. The package exists in Katello but is not in the pulp directory.



I get a 404 on this RPM.

However, it does exist in katello itself, has worked with this very same Content View in the past and works with other clients registered through a Smart Proxy. The clients currently having an issue are registered to the main katello server itself.

Expected outcome:

That a package that is in a Content View can be installed and that we don’t get 404’s when visiting the pulp directory where it is hosted out of.

Foreman and Proxy versions:

Foreman 3.1.3 and Katello 4.3.1

Foreman and Proxy plugin versions:

Distribution and version:

Rocky 8

Any ideas on what may be causing this? It is rather perplexing. Thank you

Here is the package in question, we can see it is indeed in Katello and has been installed on thousands of hosts. Currently this package can be installed if the client is registered thru a Smart Proxy (using the same exact CV in fact) but if a client is registered to the main katello server itself, we get a 404.

Here, through a Smart Proxy, we get a 200 on the same RPM:


Here it shows the package as available from the main server itself (visiting the pulp URL in a browser) but if I click it, I get a 404. Same error as when a client tries to install it via YUM.

Hopefully someone can shine some light on this

Hi @eledor ,

One note – Katello 4.3 is quite old at this point, we’re currently releasing Katello 4.11. We consider the newest two releases to be the ones “in support”.

Anyway, what content mirroring type and what download policy are you using for your repository? Is it possible that you used “on-demand” but the RPMs no longer exist in the upstream repository?

If your smart proxy has the RPMs I’m guessing they were on disk at one point, so maybe you used reclaim space to delete the old RPMs?

If that’s not it, I would recommend trying to “republish” the content view repositories. I can’t remember if Katello 4.3 supports it, but the hammer command would be hammer content-view version republish-repositories ...

If hammer doesn’t support it it should be in the API docs.

1 Like

hi @iballou

Thanks for replying to me.

For the CentOS 7 base repo, where we are pulling from the, the Download Policy is Immediate the Mirroring Policy is Additive.

Also, I checked the upstream repo where we are pulling from and that exact package is still up there:



I have tried republishing the existing Content View along with creating a brand new one with the same repos added to it and I am getting the same thing.

Any other suggestions would be great.

Thank you

hi again @iballou

b00m! That was it:

hammer content-view version republish-repositories --id 3081 --force true

(i had previously just tried republishing the CV from the GUI but this hammer command did the trick).

any ideas on what may have caused this?

also, is there a way i can run this safely on the server for all CV’s? Right now I am doing it per CV version.

per CV, regardless of version, would be ideal.



1 Like

Obviously I can’t be sure, but the fact that a “republish repository metadata” action fixed it, along with the fact that you could see the package being served, but got a 404 when trying to download it, suggests that the Pulp publication included an artifact reference to a missing file. Perhaps something was cleaned up in error. The hammer command that fixed it causes Katello to ask Pulp to re-create the Pulp publication that is served, which clearly fixed the issue.


Glad it worked! We only have it available per content view version. It is odd that it happened across much of your system. I can only guess that Pulp erroneously cleaned up some artifacts during orphan cleanup.

Hi @quba42 and @iballou

Unfortunately, this issue continues to happen and perplex us. Any more insight you may be able to provide would be most welcome.

What I can confirm is the following:

  1. Publishing a new Content View will trigger this every time. This used to not be the case.

  2. We will get the 404 error of a package that shows as available from the URL.

  3. Running republish-repositories with hammer on the Content Views fixes this until a new CV is published. Therefore, I don’t think it is a orphan cleanup triggering this?

  4. The repmod.xml file after the CV publish is old and a yum install produces these kinds of errors:

[19:33:32 root@memcache03 ~]# yum install vim
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager, tsflags, versionlock
Loading mirror speeds from cached hostfile
Cloud_SRE_CentOS_7_CentOS_7_base_x86_64 | 2.3 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_base_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:45 2023
Downloaded: Fri Aug 5 08:47:39 2022
Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64 | 2.0 kB 00:00:00
Not using downloaded Cloud_SRE_CentOS_7_CentOS_7_extras_x86_64/repomd.xml because it is older than what we have:
Current : Wed Nov 29 23:26:33 2023
Downloaded: Fri Jul 7 06:31:31 2023

Running a yum clean all on the client will make these errors go away but I think we are just using the old repmod.xml at that point. Republishing the repos will update this repmod.xml and then the clients are happy.

  1. Our current mirroring policy is “Additive”. Could this be part of the issue? Do you think it wise to change it to Content Only and can this be safely done?


  • Eledor

Some more info–

  1. This is only happening for our CentOS 7 Repos. Rocky 8 does not exhibit this behavior so it is not system wide.

  2. Republishing a CV and then trying to install a package through a Smart Proxy (not directly registered to the main katello server) we don’t have this problem. The problem only manifests itself for CentOS 7 Content Views and hosts directly registered to the main server.


This does not make any sense to me. “Running republish-repositories with hammer on the Content Views” just re-runs one of the steps that is performed when Publishing a new Content View (at least on a current version). It is inconceivable to me that this should result in different results, unless the very old version you are using contains bugs that have since been fixed.

The truth is that you are unlikely to find anyone who is going to help you with this for Foreman 3.1/pulpcore 3.16. My sense is that it would take a Programmer well versed in both Katello and Pulp (with access to the system) several hours to have a good shot at debugging this.

Yeah like @quba42 said we’ve had improvements in our self-healing since Katello 4.3. It would be a lot easier to help out if you’re able to upgrade to at least Katello 4.9. The code your own won’t be receiving any more fixes, but if you upgrade to newer code and that still breaks, it makes it a lot easier to give priority.

Something must be corrupt in your DB that’s confusing our CV publishing code I’m assuming. To tell for sure we’d need to poke around the database.

In the foreman console (foreman-rake console) you can do things like:

# List repositories linked to published repositories
repos = ::Katello::ContentViewVersion.find(<id>).repositories.archived

#  Then, for example:

# Show pulp version_href

# Show pulp publication_href

# show pulp distributions

I’m not sure how much Pulp experience you have, but it you follow this workflow guide it’ll teach you what each of those entities mean: Create, Sync and Publish a Repository — Pulp RPM Support 3.24.0 documentation

And it might help you figure out what Katello is orchestrating incorrectly.