Trying to run a yum update from a client system gives errno 14 https error 404 not found

Problem: Have an initial content-view that has been published and promoted to the first environment. In the gui, I can navigate to the content-view and can see rpms in it. On the client, I run a yum update zsh for example and it shows an incremental update is available and asks if this is OK? I enter yes and then it gives this error. I downloaded the debug certificate and loaded it into firefox and it displays 404: not found. Not sure what I have done wrong? hammer ping shows eveything is up and running. firewalld is allowing services RH-Satellite-6 dhcpv6-client ntp ssh.

Expected outcome: yum update works

Foreman and Proxy versions: 2.1.4-1

Foreman and Proxy plugin versions:

Distribution and version: CentOS 7.8

Other relevant data:

I can browse to the satellite server Packages/t and see that tuned for example is there. I can go in through the gui and look at that content-view, versions, , rpm Packages and can see it there. Just not sure why when the client system tries to dowload it I get a 404 error?

Does anyone have an idea?

Does yum show a url its trying to fetch when it gets the 404?

Yes. I can clink on that and tell my OS to open the link. In the browser it also displays 404 not found. If I then edit the URL to remove the package name and just list all rpms that begin with that letter, I can scroll down and see the rpm in that view. It is quite odd. This happens for several rpms.

Oh so its throwing a 404 for the actual RPM. Can you tail /var/log/messages and see if there is anything interesting while you see the 404?

Unfortunately, no. Nothing in messages. I even tried to increase the verbosity of yum, but that did not give any clues.

Could this be your issue? Content proxy not listing to port 80 anymore

Try disabling “Publish via HTTP”. That definitely helped me although I’m still seeing 1 random package not download on 1st try and works on 2nd try…but never the same package…

I would not think so? I’m not using a content proxy. I’ll take a look though.

I could be wrong but I think you have no choice but to use a content proxy. You’re just using the one that’s built into Foreman.

I just updated to from v2.3.1 to v2.3.2 and haven’t been able to reproduce the error again (but I had previously disabled publishing via HTTP).

1 Like

@bradawk lets try this, could you run these two commands on the foreman server:

tail -f /var/log/httpd/*

journalctl -f -u pulpcore-content

and then try to yum install something on a client?

Could you also provide the url (hostname/sensitive info marked out) that allows you to browse the repo?

1 Like

The URL is https://server/pulp/repos/MYORG/test/full_content/custom/el7_repos/base_x86_64/Packages/t/tuned-2.11.0-8.el7.noarch.rpm
If I take it back to Packages/t/ then it gives me a listing and I see the tuned rpm there.

The tail on /var/log/httpd showed:
ip - - [date] “GET /pulp/repos/MYORG/test/full_content/custom/el7_repos/base_x86_64/Packages/t/tuned-2.11.0-8.el7.noarch.rpm HTTP/1.1” 404 14 “-” “urlgrabber/3.10 yum/3.4.3”
The journalctl command gave the same error. I do notice the error message is referencing HTTP whereas I have the repo file set up with https connections. In my browser, if I try to change it to http, it automatically gets re-directed to https.
Another point is, if I take the URL back to Packages/t and then scroll down until I see the tuned rpm listed, once I click on that, I again get the 404 not found.

Out of curiosity is this repo using “On Demand” or “immediate” as its download policy?

I noticed that upstream centos has now removed 2.11.0-8 in favor of 2.11.0.9. The way “On Demand” works is that it tries to fetch it from the upstream repository, if that package is no longer there, you’ll end up with a 404. It looks like your using a snapshot within a content view, which means that if nothing had requested 2.11.0-8 before it got removed, the file wouldn’t have been downloaded and would need to be fetched from the upstream repo, but it no longer exists. We changed the default value for the download policy to ‘Immediate’ a couple of releases ago due to this issue.

If you are using ‘on demand’, i would recommend you:

  1. switch the repo to immeidate
  2. re-sync the repo
  3. republish your content view
  4. repromote it to your content view

I ran into a similar 404 error with yum history undo. I was specifically calling for old packages that appeared to be present but I believe it was likely on the meta data that was present, not an actual rpm. Maybe you have a package dependency that is calling for an older version that has been archived. If so, this solution I found below might work for you. It certainly solved my 404 pains. I don’t have a complete understanding of why the download policy didn’t download all the rpms in my case other than maybe someone switched the policy up after packages were removed leaving behind some stale repo data that needed to be regenerated. I believe you can republish that repodata, maybe that would help clear up what is actually there.

What I found was that in my environment only the upstream repo was published to the latest content views. The download policy was set to download and keep all packages but I’m not sure if someone had it set that way from the start. The result was being able to browse the packages yum desires BUT if I click them or curl to download them, they would just download the text ‘404 not found’. So the rpm was not actually there, likely just the metadata.

What fixed this was finding where the actual repositories have been moved to. For epel there is an archive location maintained by fedora. For Centos they’d go to the centos vault. I created a foreman repository for each prior release that I thought I may need ( .eg 7.7, 7.8 ). This time I set the policy to on demand thinking that these packages may remain in the archive upstream url for longer that I’ll need. Then its the normal create a new version of your content view and promote it to the lifecycle that your server is in. I did have to enable them via subscription manager on the server. After that yum was able to find the specific package version that exists in it’s new upstream location, download and install it.

Epel
https://archives.fedoraproject.org/pub/archive/epel/7.8/x86_64/

CentOS Updates
https://vault.centos.org/7.7.1908/updates/x86_64/

CentOS Base
http://mirror.centos.org/centos/7/os/x86_64/

CentOS Base Vault ( for archived packages )
https://vault.centos.org/7.7.1908/os/x86_64/

CentOS Extras
https://vault.centos.org/7.7.1908/extras/x86_64/

1 Like