Unable to install / update packages from RHEL 7 Server repository

Support
1

Problem:
Unable to install / update any package from the RHEL 7 Server repository
Expected outcome:
Package installs / updates from content view
Foreman and Proxy versions:
Foreman 2.3.3
Foreman and Proxy plugin versions:
Katello 3.18.2
Distribution and version:
CentOS 7.9
Other relevant data:
All other repositories are working just from CentOS 7,8 and other RHEL 7,8 so it is just the main RHEL 7 Server repo. This is happening on both of the configured organizations. I have re-created all the RHEL 7 repositories and content views from scratch multiple times, but still receive the same error message below.

[root@host ~]# yum -y update sos
Loaded plugins: enabled_repos_upload, langpacks, package_upload, product-id, search-disabled-repos, subscription-
: manager
ORG_Foreman_Foreman_Client_Latest_el7_x86_64 | 2.3 kB 00:00:00
ORG_Zabbix_Zabbix_5_0_el7_x86_64 | 2.0 kB 00:00:00
rhel-7-server-rpms | 2.6 kB 00:00:00
Resolving Dependencies
→ Running transaction check
—> Package sos.noarch 0:3.9-4.el7_9 will be updated
—> Package sos.noarch 0:3.9-5.el7_9.4 will be an update
→ Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================
Package Arch Version Repository Size

Updating:
sos noarch 3.9-5.el7_9.4 rhel-7-server-rpms 541 k

Transaction Summary

Upgrade 1 Package

Total download size: 541 k
Downloading packages:
No Presto metadata available for rhel-7-server-rpms
sos-3.9-5.el7_9.4.noarch.rpm FAILED
https://katello.example.com/pulp/repos/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/os/Packages/s/sos-3.9-5.el7_9.4.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

If above article doesn’t help to resolve this issue please open a ticket with Red Hat Support.

Error downloading packages:
sos-3.9-5.el7_9.4.noarch: [Errno 256] No more mirrors to try.

Each server has it’s release set to 7Server and this has been working since Jan/Feb when I first built and started using this Katello environment. The metadata seems to be found, but the failure is when the package is attempted to be downloaded for installation.

Thanks for any advice / assistance you can provide.

Cassuis242

2

The Red Hat repositories are all using the default ‘on demand’ policy for downloading content. I have also attempted to ‘immediate’, but the download will never finish successfully. I either get a ‘server disconnected’ or SSL error message during the sync process.

3

Hi @Cassius_Clay

Can you try the following:

  1. Republish repository metadata for the problem repo: Content > Products > (product name) > (RHEL 7 Server repository name) > Republish Repository Metadata
  2. If that doesn’t help, also try refreshing the manifest: Content > Subscriptions > Manage Manifest > Refresh

Let us know if one of those helps.

4

Thanks for the reply @jeremylenz.

I have tried both of your suggestions, but the issue has continued. I am able to install / update packages from repositories like RHEL 7 Extras, but just not the main Server 7 repo. I am also able to install / update packages from all three enabled RHEL 8 repos (AppStream, BaseOS, Supplementary, etc) just fine. It seems it is just the one RHEL 7 Server repository.

5

hmm, I wonder if an advanced sync would help?

Products > (product name) > Repositories > (RHEL 7 Server repository name) > Select Action button > Advanced Sync > Validate Content Sync

6

I have tried that as well which all checks out as ‘green’. My guess is there is an issue pulp3 which is causing this, but that is just a guess. I have had a lot of issues with pulp3 so far since building new on 3.18 using pulp3. I get a lot of 501 gateway errors which I believe is that pulp is too busy to respond, but that can be with only one system trying to pull packages and when having several hundred systems pulling at once seems to work well most of the time. Not sure where to go from here.

Thanks for the suggestions.

7

For the gateway errors, we have a number of fixes in Katello 4.0 that aimed at solving these issues. For example, a prominent one that handles most of the gateway errors is: Fixes #31835: Add disablereuse=on for pulpcore-content service ¡ theforeman/puppet-pulpcore@fd47fe1 ¡ GitHub

For the RHEL 7 content issue, are you syncing with on demand or immediate?

8

Hi @ehelms,

Are the fixes for the gateway errors applicable to Katello 3.18? All of the Red Hat repos are configured for on demand currently. I have tried to switch them to immediate; however, I cannot seem to get a completed sync with that download policy. It will either error out with a server disconnected or some SSL message.

9

The gateway errors are applicable to 3.18 and you could try manually applying the disableonreuse.

What are the disconnected and SSL error messages you are getting when trying to sync from the CDN?

10

Where is the correct place/way to implement the new disableonreuse option?

I changed the RHEL7 server repository to immediate and ran another sync. It ran for about 30 minutes than errored out with this message.

Response payload is not completed

That is a new one that I haven’t seen before.

11

Here is the SSL error I receive during a complete sync using the immediate policy.

Cannot connect to host cdn.redhat.com:443 ssl:default [None]

12

Inkedkatello-sync_LI
Inkedkatello-sync_LI1485×566 80 KB

13

@ehelms perhaps I’m not completely removing all details about the repo I’m having trouble with. Could you share the correct way to removing a Red Hat repo like it never was there so I could start over from scratch to see if that helps?

Thanks

14

What I have been doing is deleting all content views that use the repo then remove it from Content > Red Hat Repositories lastly run foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null 2>&1.

15

That sounds about right. @katello team may need to pop in and take a look at what could be causing this issue.

16

Hi @katello folks. I don’t mean to be pushy, but curious if you had thoughts around what I have been experiencing. I have completely removed all RHEL 7 repositories from Katello and started them over several times with the same issue as the outcome. I’m all out of ideas and stuck with no ability to deploy RPMs to RHEL 7 systems via Katello at this time.

Thanks in advance for any suggestions.

17

I suspect your immediate downloads are failing due to some sort of rate limiting on the server. You can try adjusting the download_concurrency by running:

hammer repository update --id=X --download-concurrency=2

where X is the id of the repository, you could also use --name=REPO_NAME --product=PRODUCT_NAME instead of specifying the id.

And then try re-syncing the repository using the immediate download policy.

As for your installation issues, i’d love to see the output of:

journalctl -f -u pulpcore*

while you try to yum install an rpm.

18

@Justin_Sherrill thanks for the response and suggestions. I haven’t tried the download_concurrency change and the Red Hat repositories are still using the default “on demand” policy at the moment. Here is the journalctl output while trying to update the sos RPM on a server.

May 25 15:59:09 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:09 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/ORG/ORG_el7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2216 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:10 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:10 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/Extra_Packages_for_Enterprise_Linux/Extra_Packages_for_Enterprise_Linux_7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2552 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:10 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:10 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/Foreman/Foreman_Client_Latest_el7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2519 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:10 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:10 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/Zabbix/Zabbix_5_0_el7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2225 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:11 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:11 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/extras/os/repodata/repomd.xml HTTP/1.1” 200 2547 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:11 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:11 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/optional/os/repodata/repomd.xml HTTP/1.1” 200 2554 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:12 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:11 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml HTTP/1.1” 200 2892 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:12 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:12 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/os/repodata/repomd.xml HTTP/1.1” 200 2554 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:16 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:16 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/os/Packages/s/sos-3.9-5.el7_9.4.noarch.rpm HTTP/1.1” 404 172 “-” “urlgrabber/3.10 yum/3.4.3”

Regards,
Cassuis

19

@Justin_Sherrill
Even with changing the download concurrency the downloading of the RPMs still fails with the policy set to immediate.

Capture
Capture1385×416 25.1 KB

20

@Justin_Sherrill
Not sure I can explain it, but it completed successfully this morning during the normal sync job. I have since changed the download_concurrency back to the default (10) and have successfully done another sync with the policy set to immediate. Once done, I was able to patch some development servers just fine so looks like my issues is fixed.

Thanks to everyone for their input and assistance. I love this product and the community has always been so helpful.

1 Like