Katello 4.1.2.1 - 404 error through content proxy due to incorrect location_href

Support
1

Problem:

There are updates available for elasticsearch, kibana and some more which I sync from https://artifacts.elastic.co/packages/6.x/yum/ and https://artifacts.elastic.co/packages/7.x/yum/

However, my clients connected to my content proxy get an error:

https://foreman-content.example.com/pulp/content/ORG/Production/centos7/custom/elastic/elasticsearch-6-x/Packages/k/kibana-6.8.18-x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found

Foreman and Proxy versions:

2.5.2/4.1.2.1 on both main server.
2.5.2 with latest pulpcore 3.14 rpms on content proxy.

Distribution and version:
CentOS 7.9

Other relevant data:

In the journal for pulpcore-content there is this error:

Aug 09 18:10:27 foreman-content.example.com pulpcore-content[55347]: pulp [None]: pulpcore.content.handler:WARNING: Could not download remote artifact at 'https://foreman.example.com/pulp/content/ORG/Production/centos8/custom/elastic/elasticsearch-6-x/6.8.18/kibana-6.8.18-x86_64.rpm': 404, message='Not Found', url=URL('https://foreman.example.com/pulp/content/ORG/Production/centos8/custom/elastic/elasticsearch-6-x/6.8.18/kibana-6.8.18-x86_64.rpm')
Aug 09 18:10:27 foreman-content.example.com pulpcore-content[55347]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://foreman.example.com/pulp/content/ORG/Testing/centos7/custom/elastic/elasticsearch-6-x/6.8.18/kibana-6.8.18-x86_64.rpm'))
Aug 09 18:10:27 foreman-content.example.com pulpcore-content[55347]: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://foreman.example.com/pulp/content/ORG/Testing/centos7/custom/elastic/elasticsearch-6-x/6.8.18/kibana-6.8.18-x86_64.rpm'))

That, of course, is the wrong URL. The correct URL on the main server is https://foreman.example.com/pulp/content/ORG/Testing/centos7/custom/elastic/elasticsearch-6-x/Packages/k/kibana-6.8.18-x86_64.rpm

The upstream repo uses the version directory. At elastic.co the URL for the RPM is https://artifacts.elastic.co/packages/6.x/yum/6.8.18/kibana-6.8.18-x86_64.rpm

For some reason, the content proxy synced the structure of the upstream repo into it’s database although it’s accessing pulp on the main server…

This is from the pulpcore database on the content proxy:

pulpcore=# select name, epoch, version, release, arch, url, location_base, location_href from rpm_package where name = 'kibana' and version like '6.8.18';
  name  | epoch | version | release |  arch  |          url           | location_base |          location_href          
--------+-------+---------+---------+--------+------------------------+---------------+---------------------------------
 kibana | 0     | 6.8.18  | 1       | x86_64 | https://www.elastic.co |               | 6.8.18/kibana-6.8.18-x86_64.rpm
(1 row)
2

Opened Bug #33241: 404 error through content proxy due to incorrect location_href - Katello - Foreman

2 Likes
3

Hi @gvde

We have triaged this issue and will start to look into it.

1 Like
4

It seems much worse. I have just tried a pxe installation of a new host which also had trouble downloading various packages. For instance with bind-libs:

access of the client to the content proxy via

https://foreman-content.example.com/pulp/content/ORG/Production/centos7/custom/centos7/updates_x86_64/Packages/bind-libs-9.11.4-26.P2.el7_9.5.x86_64.rpm

(which is listed in the Packages/ directory view)

generates this access to the main server:

Aug 20 08:16:18 foreman-content.example.com pulpcore-content[36059]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://foreman.example.com/pulp/content/ORG/Production/centos7/custom/centos7/updates_x86_64/Packages/b/bind-libs-9.11.4-26.P2.el7_9.5.x86_64.rpm'))
Aug 20 08:16:18 foreman-content.example.com pulpcore-content[36059]: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://foreman.example.com/pulp/content/ORG/Production/centos7/custom/centos7/updates_x86_64/Packages/b/bind-libs-9.11.4-26.P2.el7_9.5.x86_64.rpm'))
Aug 20 08:16:18 foreman-content.example.com pulpcore-content[36059]: pulp [None]: pulpcore.content.handler:WARNING: Could not download remote artifact at 'https://foreman.example.com/pulp/content/ORG/Production/centos7/custom/centos7/updates_x86_64/Packages/b/bind-libs-9.11.4-26.P2.el7_9.5.x86_64.rpm': 404, message='Not Found', url=URL('https://foreman.example.com/pulp/content/ORG/Production/centos7/custom/centos7/updates_x86_64/Packages/b/bind-libs-9.11.4-26.P2.el7_9.5.x86_64.rpm')

The main server doesn’t use the initial letter directory, i.e. the correct URL would be:

https://foreman.example.com/pulp/content/ORG/Production/centos7/custom/centos7/updates_x86_64/Packages/bind-libs-9.11.4-26.P2.el7_9.5.x86_64.rpm

So neither the main server nor the content proxy use this initial letter subdirectory for bind-libs. It’s only on the external centos mirror…

5

The pulpcore database on the content proxy references the b subdirectory:

[root@foreman-content ~]# sudo -u postgres psql pulpcore
pulpcore=# select content_ptr_id, name, epoch, version, release, arch, url, location_base, location_href from rpm_package where name = 'bind-libs' and version like '9.11.4';
            content_ptr_id            |   name    | epoch | version |    release    |  arch  |                url                | location_base |                    location_href                     
--------------------------------------+-----------+-------+---------+---------------+--------+-----------------------------------+---------------+------------------------------------------------------
 50b244e3-023f-4f21-8a35-a87425d43ebe | bind-libs | 32    | 9.11.4  | 26.P2.el7     | i686   | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7.i686.rpm
 78a08b09-dd87-4383-9a5a-647b9c643dc4 | bind-libs | 32    | 9.11.4  | 26.P2.el7     | x86_64 | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7.x86_64.rpm
 3457a2d3-3804-4cb0-8c78-f9f2e6fae40a | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.2 | i686   | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.2.i686.rpm
 5e488273-931c-42ee-b8d2-fe1ed0fe4d8d | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.2 | x86_64 | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.2.x86_64.rpm
 3f1d5b16-2938-4085-a915-b1790ea030da | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.3 | i686   | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.3.i686.rpm
 8e5251cc-cf45-4542-8752-5015aa482465 | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.3 | x86_64 | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.3.x86_64.rpm
 867bfda6-55ae-4548-bb79-ab868231e640 | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.4 | i686   | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.4.i686.rpm
 31b4135c-aab4-430a-9b86-c44583082ebd | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.4 | x86_64 | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.4.x86_64.rpm
 f091922d-988f-4c7d-8b45-5a44df91a078 | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.5 | i686   | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.5.i686.rpm
 60ccb323-f17d-4ed5-8b87-c08e38c89a43 | bind-libs | 32    | 9.11.4  | 26.P2.el7_9.5 | x86_64 | http://www.isc.org/products/BIND/ |               | Packages/b/bind-libs-9.11.4-26.P2.el7_9.5.x86_64.rpm
(10 rows)

But that location_href is only correct on the main server to download from the external centos 7 mirror.

So it seems to me pulp tries to maintain the external repository directory structure but that doesn’t always work and thus the content is provided in a different structure on the main foreman server.

The content proxy seems to pick up the location_href from the external server and simply copies it, even though it’s not correct…

6

Also affected.
At least, I didn’t need days to track it down this time…

7

For some repositories it’s pretty badly broken. For instance, I sync the foreman 2.5 repository to the content proxy. I used ‘wget -r’ to check what’s available: from 686 files only 16 were available for download, i.e. 670 broken links.

On the content proxy, it uses Package/f/ etc. sub-directories. On the main server it’s all in the main directory:

Main server: /pulp/content/ORG/Production/centos7-epel7/custom/foreman/2_5_el7_x86_64/

config.repo
foreman-2.5.0-0.8.rc2.el7.noarch.rpm
foreman-2.5.0-0.8.rc3.el7.noarch.rpm
foreman-2.5.0-1.el7.noarch.rpm
foreman-2.5.1-1.el7.noarch.rpm
foreman-2.5.2-1.el7.noarch.rpm
...

Content proxy:

    Packages/
    config.repo
    repodata/

The proxy tries URLs like this against the main server:

pulpcore.content.handler:WARNING: Could not download remote artifact at 'https://foreman.example.com/pulp/content/ORG/Production/centos7-epel7/custom/foreman/2_5_el7_x86_64/Packages/t/tfm-scldevel-7.0-2.el7.x86_64.rpm': 404, message='Not Found', url=URL('https://foreman.example.com/pulp/content/ORG/Production/centos7-epel7/custom/foreman/2_5_el7_x86_64/Packages/t/tfm-scldevel-7.0-2.el7.x86_64.rpm')

In other words: it’s totally messed up…

8

Is this really only going to be fixed in Katello 4.3 (which would be Foreman 3.2, if I’m not mistaken)?

9

I’m going to look into this on Monday

2 Likes
10

@gvde Could you provide me a copy of repomd.xml and {*}primary.xml.gz for both the upstream repos you are having issues with and the content proxy? I cannot seem to access https://artifacts.elastic.co/packages/7.x/yum/ or any variants such as …/7.8/yum/, it just brings up a blank page.

11

Yes. Browsing is disabled. repomd.xml is in the repodata subdirectory: https://artifacts.elastic.co/packages/7.x/yum/repodata/repomd.xml

And note, it also affects other repositories like my local sync of the foreman release repository. The elastic/kibana was only the first instance I have noticed the issue.

12

I suspect pulp3 migration to be one source if this issue. Didn’t pulp2 always use the same directory layout for the repositories? pulp3 seems to try using the upstream layout instead and thus uses that on artifacts synced with pulp3.

I guess another scenario might be if upstream changes the layout at some point and reorganizes the whole repository…

But those are just two wild guesses. The worst at the moment is that there is no workaround. I have tried any kind of upstream sync through the gui and did a couple of full syncs of the content proxy, nothing fixes it…

13

I have this on a very clean 2.5.x/4.1.y install. No migration involved.

You can probably reproduce it if you install 2.5.3, sync CentOS 7 and the foreman repos, install a proxy and try to install CentOS 7 off that proxy.

14

I have updated to pulp_rpm 3.14.3. Today there came updates for perfsonar and none of the updates were downloadable. Metadata pointed packages to subdirectory RPMS which doesn’t exist. This was a problem on the main server as well as the content proxy.

It seems I have to regenerate the metadata for my repository. That seems to have fixed it in the Library.

Is there a hammer command to regenerate the metadata for a repository so that I can script that for all? It seems a reasonable thing to do…

15

This is a total disaster. Now I have regenerate metadata and the metadata fits to the packages on the server. But now then content proxy again tries to access the rpms on an incorrect URI even after a complete sync…

16

@gvde The only thing that matters is the “location_href” written in the metadata, and the actual location of the files as they are published by Pulp. The “location_href” information via the content APIs is misleading because Pulp doesn’t use it when syncing and it doesn’t use it when publishing - really this field should be called “filename” because that is the only portion of it that is actually being used. Eventually we’ll rename the field, because the extra information is useless to us, and misleads users.

Just to give you an update, I’ve been trying to reproduce this and having difficulty doing so. Please let me know - is your “main” server and content proxy both using mirror-sync, or is one doing so and one is not? And if - on the server where you believe the incorrect metadata is originating from, you flip the value of “mirror” to the opposite of whatever it currently is, do you still encounter issues?

17

And if - on the server where you believe the incorrect metadata is originating from, you flip the value of “mirror” to the opposite of whatever it currently is, do you still encounter issues?

After a subsequent sync, because obviously just flipping the value won’t do much.

18

I don’t know what you mean exactly. My main server is doing mirror on sync on all repositories. My content proxy is doing “on demand”, if that’s what you mean. But I have never really touched that.

The metadata seems correct in either place (after I have generated the metadata for my perfsonar repository in this last case). primary.xml points to the urls listed in Packages on the main server as well as the content server. The problem is just, that the content server is using different URLs to access the main server.

So the upstream repo for perfsonar is Index of /rpms/el7/x86_64/main

After the metadata regeneration and complete content sync, both main server and content proxy present the packages in the Packages/?/ directories with ? begin the first letter of the rpm.

However, the content proxy uses URIs like “packages/pscheduler-tool-ethr-4.4.1-1.el7.noarch.rpm” to access RPMs from the main server. So, for whatever reason it is using “packages” instead of “Packages” and doesn’t use the first letter subdirectory.

It’s unclear to me where the content proxy picks up those “prefixes” to the rpms.

I can’t really tell when it’s broken and when not. I suspect, one reason could be if at some point the external repository changes it’s directory layout for the rpms. Pulp tries to “mimic” this layout in its own content presentation and if it changes externally, it results in confusion. It’s also unclear to me, under which circumstances pulp_rpm mimics the layout of the external repository and when not. I have some repositories, where pulp presents the content simply in “Packages/?/” even though the external repositories uses a different layout. For other repositories, like the kibana it uses (or used) the external layout.

I think pulp_rpm 3.14.3 also changed something in the layout. From my observations, it seems to prefer the “Packages/?/” layout now… If that’s true, it would further complicate things because now after updating to pulp_rpm 3.14.3 the layout changes through the system (after a metadata regeneration at least, I guess).

19

That question is still open. Either I look in the wrong place or hammer doesn’t support it…

20

There was no change, so I’m not sure what’s going on there. But I do now have some ideas about what could be causing the issue. Let me give some background.

Katello has an option called “mirror on sync”. This provides a flag “mirror=True” to Pulp. When syncing in this mode, Pulp adopts the behavior of pulling down an exact copy of the metadata, and making the packages available in the exact same locations they were in at the upstream. It doesn’t do a real “publish”, it basically just clones the repo.

If you regenerate the metadata, this is a manual publish, which will use a standardized layout - packages get put into Packages/${letter}/${full_name}.

When you do an ‘on_demand’ sync, Pulp stores the URLs of the artifacts so that they can be downloaded in the future.

It seems like at any given time the repo metadata on both systems is actually correct, but one or perhaps several of these things is happening:

  • The content proxy is being synced and then afterwards the layout of the repos on the main system changes, invalidating the stored URLs, and the content proxy isn’t being re-synced after that
  • The content proxy is being synced against the correct layout / both layouts, but isn’t updating the URLs on sync as it is supposed to (multiple URLs are stored in order for the same artifacts, they get tried in-order at download time)
  • The content proxy is getting updated with the new URLs, but something is going wrong when it tries the various ones.

I guess another scenario might be if upstream changes the layout at some point and reorganizes the whole repository…

So that basically is what is happening, but the “upstream” that is being reorganized is the main instance, when you alternate between syncing and regenerating the metadata.