Status code: 403 for Error: Failed to download metadata for repo after upgrade to 3.11

Hello community

after upgrade from Foreman 3.10.0 and Katello 4.12.1 to Foreman 3.11 and Katello 4.13, I got an error message when a new client tries to download repolist or search for packages inside the rep.
Error: Status code: 403 for …
Error: Failed to download metadata for repo …
I followed this documentation Upgrading Foreman to 3.11

dnf search tmux
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 778 B/s | 73 B 00:00
Errors during downloading metadata for repository ‘rhel-9-for-x86_64-baseos-rpms’:
- Status code: 403 for https:///pulp/content/Orga/RHEL_ENV/rhel_92_cont_view/content/dist/rhel9/9.2/x86_64/baseos/os/repodata/repomd.xml (IP: 172.17.190.5)
Error: Failed to download metadata for repo ‘rhel-9-for-x86_64-baseos-rpms’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

can anyone help me please?

best regards
T. Reineck

1 Like

This is usually resolved by refreshing your manifest. In the web UI: Content > Subscriptions > Manage Manifest > Refresh

Hello,

I refreshed the manifest. But the error also appears.

best regards
Tob. Rein.

Hi,
I have got the same issue with my instance. This instance is also freshly upgraded from 3.10 (I cannot confirm that it worked with 3.10).

I have refreshed the manifest. I have no problems viewing the content in a browser with the debug certificate.
Trying to curl using the entitlement cert and key gives the following error:
“403: Requested path is not a subpath of a path in the client certificate.”

Does anyone have any ideas? Thank you!

Kind regards

For folks having this issue, can you please share the ‘authorized content URLs’ and ‘Content’ sections from rct cat-cert /etc/pki/entitlement/<number>.pem ?

It’s looking like the allowed URL in the entitlement cert might be misconfigured, at least on a reproducer we found.

Also, as a workaround, try subscribing your hosts to Default Organization View / Library.

1 Like

This is appearing to be a regression with Candlepin 4.4, but an investigation is ongoing. There are escape characters erroneously showing up in the entitlement certificates’ content URLs. Candlepin 4.3 should work fine with Katello 4.13 if people are able to do a reset to downgrade. You’d just need to change the candlepin repo definition to use the 4.3 branch. Regardless, we’ll make sure to get it fixed as quickly as possible.

1 Like

exactly same here, we updated today:

rpm -q foreman katello candlepin
  foreman-3.11.0-1.el8.noarch
  katello-4.13.0-1.el8.noarch
  candlepin-4.4.10-1.el8.noarch

only rew-rhel (registered after upgrade) systems are affected.
existing systems can pull content.

Also, as a workaround, try subscribing your hosts to Default Organization View / Library.

I can confirm this workaround succeeds, if i switch to Library, it starts pulling again.

If we can help with some testing and debugging, just shout

Thanks Chris

This makes sense, as the existing ones have certificate issues by the old (4.3) Candlepin, which doesn’t have that bug.

2 Likes

ahhh, that explains a lot!
I will try to downgrade candlepin to 4.3 and come back with the documentation how to do it.
Do you know if its safe from database perspective (db scheme)?
Thanks for answering. Chris

I would not expect it to be safe to downgrade, but I am also not an authoritative source for that :wink:

1 Like

Subscribing the hosts to Default Organization View / Library does also work on my instance.

2 Likes

It’s tracked in a private RH Jira issue.
I’ve asked if it can be made public, but have no idea what the Candlepin project policies are.

1 Like

but I am also not an authoritative source for that :wink:

you are, after doing more tests, i can confirm katello does not work with downgraded candlepin-4.3, so we have to restore or wait for fix.
thanks for the support!

1 Like

What most probably should work is: take a 4.12 Katello (with CP 4.3) and before upgrading it to 4.13 (but with 4.13 repos already present) re-configure the repos to point at CP 4.3 ones, so you never upgrade/migrate to 4.4.

1 Like

2297301 – Incorrectly url encoded 'Authorized Content Url' in SCA certificates is a public BZ you can watch :slight_smile:

3 Likes