Clients cannot download certain packages from Katello/Pulp

Support
1

Problem:
From time to time I’ve run into the following issue where clients of my can not download certain packages from Katello/Pulp resulting in a 404 error not found. I don’t quiet understand why that is the case as clients can download other packages from the same content view or even same repo. If I check the content view from the GUI I can see the package/s being listed.

In the past what usually got it working for me is manually syncing the repos and creating a new content view version. However in the current case that’s not helping and I’m not sure what else I can try out here. Any thoughts?

Foreman and Proxy versions:
foreman: 3.11.0

Foreman and Proxy plugin versions:
foreman-tasks 9.1.1
foreman_puppet 6.3.0
foreman_remote_execution 13.1.0
katello 4.13.0

Distribution and version:
Rocky Linux release 8.10 (Green Obsidian)

Other relevant data:

[root@localhost thrall]# dnf update -y
...
[MIRROR] ffmpeg-5.1.6-1.el9.x86_64.rpm: Status code: 404 for https://foreman.my.company.com/pulp/content/MyCompany/Production/RockyLinux9All/custom/RockyLinux9/RockyLinux9RPMFusion/Packages/f/ffmpeg-5.1.6-1.el9.x86_64.rpm (IP: 10.1.10.10)
[MIRROR] ffmpeg-5.1.6-1.el9.x86_64.rpm: Status code: 404 for https://foreman.my.company.com/pulp/content/MyCompany/Production/RockyLinux9All/custom/RockyLinux9/RockyLinux9RPMFusion/Packages/f/ffmpeg-5.1.6-1.el9.x86_64.rpm (IP: 10.1.10.10)
[MIRROR] ffmpeg-5.1.6-1.el9.x86_64.rpm: Status code: 404 for https://foreman.my.company.com/pulp/content/MyCompany/Production/RockyLinux9All/custom/RockyLinux9/RockyLinux9RPMFusion/Packages/f/ffmpeg-5.1.6-1.el9.x86_64.rpm (IP: 10.1.10.10)
[MIRROR] ffmpeg-5.1.6-1.el9.x86_64.rpm: Status code: 404 for https://foreman.my.company.com/pulp/content/MyCompany/Production/RockyLinux9All/custom/RockyLinux9/RockyLinux9RPMFusion/Packages/f/ffmpeg-5.1.6-1.el9.x86_64.rpm (IP: 10.1.10.10)
[FAILED] ffmpeg-5.1.6-1.el9.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
  ffmpeg-5.1.6-1.el9.x86_64: Cannot download, all mirrors were already tried without success
Package action failed, exiting...
Exit status: 1

[root@localhost]# wget --no-check-certificate https://foreman.my.company.com/pulp/content/MyCompany/Production/RockyLinux9All/custoges/f/ffmpeg-5.1.6-1.el9.x86_64.rpm
--2024-09-12 09:14:28--  https://foreman.my.company.com/pulp/content/MyCompany/Production/RockyLinux9All/custoges/f/ffmpeg-5.1.6-1.el9.x86_64.rpm
Resolving foreman.my.company.com (foreman.my.company.com)... 10.1.10.10
Connecting to foreman.my.company.com (foreman.my.company.com)|10.1.10.10|:443... connected.
WARNING: The certificate of ‘foreman.my.company.com’ is not trusted.
WARNING: The certificate of ‘foreman.my.company.com’ doesn't have a known issuer.
HTTP request sent, awaiting response... 404 Not Found
2024-09-12 09:14:29 ERROR 404: Not Found.
2

Does the content view use filters? Perhaps the package was filtered out.

If you change the host to use Library / Default Organization View, do you still have the problem?

3

This content view doesn’t have any filters applied. I also just tried again after changing the host to use the Library View, but I’m still seeing the same issue after running a subscription-manager refresh and dnf clean all.

4

You could try running a ‘Complete’ or advanced sync on the repository. It’s possible there’s some sort of corruption or irregularity there.

5

Thanks Jeremy. I ran an advanced sync on the repo and then created a new version on my content view. However I’m not able to get any packages from that repo now. Other repos are fine as far as I can tell. Is there a way to easy locate the repo packages on the server?

6

This seems like a problem with the upstream repo. I don’t know of a way to resolve it other than to contact Rocky or whoever maintains that.

1 Like
7

Ok thanks Jeremy. I’ll see what I can do :slight_smile:

8

For anyone running into the same issues here’s what I finally figured out.
We have a local mirror server which runs /bin/dnf reposync -g --delete -p /path/to/storage --repoid=appstream-8 --newest-only --download-metadata every night. The --newest-only flag only downloads the latest version of each package but the metadata on that repo still contains the old version inside the metadata file list. When syncing repos within foreman/katello itl complains that those older package versions can’t be located. So our workaround right now is that we update the local repos after the dnf reposync via /bin/createrepo_c --update /path/to/repo.