Synced repository not available (404 error)

[andyfry]

Problem:
Not sure if I’ve missed something but I’m trying to synchronise repositories on a fresh kaetllo 3.16 disconnected server. The sync from my local repositories works fine and completes successfully (after much jiggery pokery) but the Published at link gives a 404 error and the content is not available even after publishing it in a content view the clients get 404 on the repomd.xml file

Whilst the details below refer to a single repository, this is effecting ALL repositories i.e. no content is available:

I’ve verified permissions, and that the files /links are in /var/lib/pulp, restarted server, republished metadata, Validated content sync. All with no joy.

Expected outcome:
Able to sync CentOS 7, CentOS 8 and EPEL repositories and publish that content.

Foreman and Proxy versions:
Installed Packages
foreman.noarch 2.1.3-1.el7 @foreman-local
katello.noarch 3.16.1-1.el7 @katello-local

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7.9

Other relevant data:
Clicking on Published at link gives the following in apache log:

9.9.9.9 - - [26/Nov/2020:14:56:00 +1030] “GET /pulp/repos/ORG/Library/custom/CentOS_8/CentOS_8_AppStream/ HTTP/1.1” 404 14 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36”

dnf install katello-host-tools on an attached client give the following in apache log:

10.0.0.0 - - [26/Nov/2020:14:56:07 +1030] “GET /rhsm/consumers/049c4a3f-a49a-4200-8a37-53aa2660cbe0/certificates/serials HTTP/1.1” 200 485 “-” “RHSM/1.0 (cmd=dnf)”
10.0.0.0 - - [26/Nov/2020:14:56:08 +1030] “GET /rhsm/status HTTP/1.1” 200 399 “-” “RHSM/1.0 (cmd=dnf)”
10.0.0.0 - - [26/Nov/2020:14:56:08 +1030] “GET /rhsm/consumers/049c4a3f-a49a-4200-8a37-53aa2660cbe0/content_overrides HTTP/1.1” 200 2 “-” “RHSM/1.0 (cmd=dnf)”
10.0.0.0 - - [26/Nov/2020:14:56:08 +1030] “GET /pulp/repos/ORG/DEV/CentOS8/custom/CentOS_8/CentOS_8_BaseOS/repodata/repomd.xml HTTP/1.1” 404 14 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
10.0.0.0 - - [26/Nov/2020:14:56:08 +1030] “GET /pulp/repos/ORG/DEV/CentOS8/custom/CentOS_8/CentOS_8_BaseOS/repodata/repomd.xml HTTP/1.1” 404 14 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
10.0.0.0 - - [26/Nov/2020:14:56:08 +1030] “GET /pulp/repos/ORG/DEV/CentOS8/custom/CentOS_8/CentOS_8_BaseOS/repodata/repomd.xml HTTP/1.1” 404 14 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”
10.0.0.0 - - [26/Nov/2020:14:56:08 +1030] “GET /pulp/repos/ORG/DEV/CentOS8/custom/CentOS_8/CentOS_8_BaseOS/repodata/repomd.xml HTTP/1.1” 404 14 “-” “libdnf (CentOS Linux 8; generic; Linux.x86_64)”

Repository view

[andyfry]

FWIW

-rw-r–r--. apache apache system_u:object_r:httpd_sys_rw_content_t:s0 /var/lib/pulp/published/yum/master/yum_distributor/6ba9ed02-6572-404b-aa05-c0a304c8a88b/1606366455.74/repodata/repomd.xml

[matemikulic]

hey, does your repo has any authentication setup on one of the files or multiple? there was a bug in katello 3.16 with pulp authentication and it as throwing 404. I’ve had that issue. However in katello 3.17, pulp team has resolved this bug and I was able to sync without issues after upgrade.

[andyfry]

Hi @matemikulic,

Nope, there is no authentication on the proxy or repos. I have had to build my own local repo because the proxy interferes with the content so on my repo server I’m doing a reposync, then rebuilding respond with a create repo.
This content then syncs nicely into my katello/pulp repo without any failure but the content it published gives the 404 errors.

[andyfry]

More information if it helps:

I created a new organisation, added centos7(base, centosplus, extras and updates) repos, ran a synchronise on all of them having configured variations of Mirror on Sync, immediate and ondemand download policies and nothing appears to have changed.
i.e. I still get 404 when I try to access the repo via get or yum.

These repos did work ok at one point but have stopped working now. I can’t say for certain but they may have stopped working after I tried to sync the centos errata using information from here:

CentOS errata, what am I missing?

Last time I checked CentOS didn’t publish erratas in their repos, that’s why you don’t see them. There are ways[1] how to get them from other source and “sideload” them into katello. Quick search found some tools to do that[2,3]. [1] - https://www.lisenet.com/2018/katello-import-centos-errata-into-pulp/ [2] - https://github.com/rdrgmnzs/pulp_centos_errata_import [3] - https://github.com/nicolas-r/katello-centos-errata-import

Is it possible that has had an effect?

[andyfry]

Or is it possibly related to me forcing use of pulp 2 ? Just been trying to track the urls which are being used here and it seems pretty complex but…
the repository is published at http://foreman/pulp/repos/Library/custom/CentOS_8/CentOS_8_BaseOS/

seems to ProxyPass to http://localhost/pulp/content but I do see definitions for /pulp2/content

Are my urls wrong somehow?

[grajangam]

@andyfry I’m having same issues after forcing use of pulp2 to have centos errata import. Let me know if you could fix these 404 errors.

[gvde]

That doesn’t really make sense. What do you mean by “forcing use of pulp2”?

Either you are still on pulp2, i.e. you did not run switchover, then there is nothing to “force”.

Or you made the switchover. Then you are on pulp3. You can’t go back. You cannot force pulp2 then either…

[Justin_Sherrill]

does it work if you use /pulp/content instead of /pulp/repos/ ?

curl " http://HOSTNAME/pulp/content/ORG/DEV/CentOS8/custom/CentOS_8/CentOS_8_BaseOS/repodata/repomd.xml"

What about the ‘library’ repo?

curl " http://HOSTNAME/pulp/content/ORG/Library/custom/CentOS_8/CentOS_8_BaseOS/repodata/repomd.xml"
curl " http://HOSTNAME/pulp/repos/ORG/Library/custom/CentOS_8/CentOS_8_BaseOS/repodata/repomd.xml"

do any of those work?

[andyfry]

@gvde ,

It was an option in the foreman-installer, but may have related to debs. This was quite some time ago and I’m now running katello 4.1 and foreman 2.5 without this problem occurring.

@Justin_Sherrill ,

I believe the content url worked but like I say, I’ve moved on.

All good