Problem:
when running dnf update on clients (running RHEL8) clients gets this error:
Errors during downloading metadata for repository ‘rhel-8-for-x86_64-baseos-rpms’:
Status code: 403 for https:/katello-server/pulp/content/Operations/Library/Patching/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml (IP: 10.x.x.x )
Error: Failed to download metadata for repo ‘rhel-8-for-x86_64-baseos-rpms’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
On Katello server this message can be seen on katello /var/log/messages :
pulpcore-content[3599]: pulp [None]: pulp_certguard.app.models:WARNING: Path /Operations/Library/Patching/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml is not allowed in client cert, some clients are working some not Expected outcome:
pulpcore-content[3467]: [02/Aug/2024:07:24:05 +0000] “GET /pulp/content/Operations/Library/Patching/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1” 200 Foreman and Proxy versions:
Foreman Version 3.11.0
Pulpcore, Logs, TFTP, Ansible, Dynflow, and Script
3.11.0 Foreman and Proxy plugin versions:
foreman-tasks 9.1.1
foreman_ansible 14.0.0
foreman_leapp 1.2.1
foreman_remote_execution 13.1.0
foreman_virt_who_configure 0.5.21
katello 4.13.0
Distribution and version:
RHEL8.10 Other relevant data:
I think this is the same problem as Status code: 403 for Error: Failed to download metadata for repo after upgrade to 3.11, which was discussed quite extensively there.
tl:dr: There is a problem with candlepin 4.4.10 (I guess that is the version you are running?). That issue should be fixed with an update to candlepin 4.4.12/4.4.13, but those versions currently struggle with other problems that are also described in that thread and for which currently only workarounds exist.
Hi Areyus
thanks for the info, yes looks so, but i tried yesterday to update from candlepin 4.4.10 to 4.4.13 , and ended up in revert all back, first the process stucked within DB, after running foreman-rake:db it finished but it was very slow, login in katello and client connections ended up in timeout by running dnf install. I can see on some our clients that some of them can reach the repos and some not, i thing clients which was registered before update to 3.11 are working. Is there any other solution for this ?
From what I understand, the root cause for your problem is that candlepin 4.4.10 generates faulty client certs, so it is expected that all hosts registered before the upgrade to Katello 4.13 would work correctly since they got correctly generated certs from the old version. As far as I understand, you basically have two options to get back to a working system asap:
Revert to Foreman 3.10/Katello 4.12, though this is only possible if you still have a working backup from before the upgrade and will most likely come with dataloss.
Upgrade to Candlepin 4.4.12 or 4.4.13 and install rng-tools and enable rngd (or any similar service for better RNG seeding). The rootcause for the problems with Candlepin >=4.4.12 are described here, where you can also read up on other people confirming that enabling rngd is a working workaround
A proper solution would probably be to wait for another release of candlepin that actually fixes the underlying issue, but candlepin 4.4.13 with rngd is probably your fastest way to get back to a working system.
