I updated to Foreman/Katello 3.16/4.18 and it seems to have broken some of our own packages in our own repo. What I noticed was that when trying to install any of a certain type of package from any client ended with error 404, “Not Found”.
What is “special” about these packages is they the filename does not match the filename inside the rpm.**
**
Ex.
In our own in house repo
rpm filename**:** SSPGateway-2.7.4-el7.x86_64.rpm
$ rpm -qip SSPGateway-2.7.4-el7.x86_64.rpm
Name : ssp-gateway
Version : 2.7.4
Release : 202210261323.el7
Architecture: x86_64
Source RPM : ssp-gateway-2.7.4-202210261323.el7.src.rpm
Hammer shows when searching the package SSPGateway-2.7.4-el7.x86_64.rpm:
--------|---------------------------------|-------------------------------------------
| ID | FILENAME | SOURCE RPM |
|---|---|---|
| 6905451 | SSPGateway-2.7.4-el7.x86_64.rpm | ssp-gateway-2.7.4-202210261323.el7.src.rpm |
| -------- | --------------------------------- | ------------------------------------------- |
Using curl to browse packages on Foreman and proxy:
# curl -s https://foreman.xxx.internal/pulp/content/xxx/Library/custom/xxxxxx_rpm/prod_el7/Packages/s/| grep -i Gateway-2.7.4
SSPGateway-2.7.4-el7.x86_64.rpm 15-Jan-2025 11:44 76.8 MB
curl -s https://fmproxy.xxx.internal/pulp/content/xxx/Library/custom/xxxxxx_rpm/prod_el7/Packages/s/| grep -i Gateway-2.7.4
ssp-gateway-2.7.4-202210261323.el7.x86_64.rpm 15-Jan-2025 11:45 76.8 MB
So far we see Foreman kept the filename of the original rpm but the proxy does not and instead using the information inside the rpm to create a filename. This has not been an issue before.
The problem is that this package are now impossible to download from the proxy by any client.
I can see dnf find the package and tries to download the filename ssp-gateway-2.7.4-202210261323.el7.x86_64.rpm but error 404.
Trying to download the package using wget fails as well with the same error. It is like there is some kind of disconnect between Foreman and the Proxy and I can not seem to understand why.
ex.
curl -s https://fmproxy.xxx.internal/pulp/content/xxx/Library/custom/xxxxxx_rpm/prod_el7/Packages/s/| grep -i Gateway-2.7.4
ssp-gateway-2.7.4-202210261323.el7.x86_64.rpm 15-Jan-2025 11:45 76.8 MB
wget https:/fmproxy.xxx.internal/pulp/content/xxx/Library/custom/xxxxxx_rpm/prod_el7/Packages/s/ssp-gateway-2.7.4-202210261323.el7.x86_64.rpm
–2025-10-28 19:42:41-- https://fmproxy.xxx.internal/pulp/content/xxx/Library/custom/xxxxxx_rpm/prod_el7/Packages/s/ssp-gateway-2.7.4-202210261323.el7.x86_64.rpm
Resolving fmproxy.xxx.internal (fmproxy.xxx.internal)… 10.10.10.10
Connecting to fmproxy.xxx.internal (fmproxy.xxx.internal)|10.10.10.10|:443… connected.
HTTP request sent, awaiting response… 404 Not Found
2025-10-28 19:42:43 ERROR 404: Not Found.
Done a lot of trouble shooting, even deleted the package from Foreman, renamed the package to ssp-gateway-2.7.4-202210261323.el7.x86_64.rpm on my repo, updated the repo, resynced the repo with Foreman, package name still shows SSPGateway-2.7.4-el7.x86_64.rpm in Foreman!!! and ssp-gateway-2.7.4-202210261323.el7.x86_64.rpm on the proxy and error 404.
Also tried to reclaim unused space, ran “foreman-rake katello:delete_orphaned_content”, ran full syncs on proxies and Foreman. Also created a new repo, added the same packages to this repo, same problem!
I am a little lost on what is going on and how the even after renaming it still shows as SSPGateway-2.7.4-el7.x86_64.rpm in Foreman. Any trouble shooting tips are welcome 
Can add that we have 1000+ of our own packages from our own repos and I only see this issue for packages where the filename on Foreman and the Proxy is different.