[Errno 14] HTTPS Error 403 - Forbidden (RedHat repositories only)

Problem:

Yum repolist/check-update fails on published redhat repositories. Other repos works.

https://x01-srv01/pulp/repos/x01/Production/CV-RHEL7/content/dist/rhel/server/7/7Server/x86_64/extras/os/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden

The following remedies has been tried and failed:

  • yum clean all
  • subscription-manager unregister/register
  • Regenerated certificates (–certs-update-ca --certs-update-server --certs-update-all)
  • Installed new katello-ca-consumer package
  • SELinux in permissive mode
  • Removed all relevant content views and repositories and readded/synced.
  • Refreshed subscription manifest

Expected outcome:

Shows number of packages and being able to install/update packages.

Foreman and Proxy versions:

Foreman 2.1.2 / Katello 3.15

Foreman and Proxy plugin versions:

Distribution and version:

CentOS 7

Other relevant data:

/var/log/httpd/foreman-ssl_access_ssl.log

“GET /pulp/repos/x01/Production/CV-RHEL7/custom/Foreman_Client/foreman_client_el7_x86_64/repodata/repomd.xml HTTP/1.1” 200 3836 “-” “urlgrabber/3.10 yum/3.4.3”
“GET /pulp/repos/x01/Production/CV-RHEL7/content/dist/rhel/server/7/7Server/x86_64/extras/os/repodata/repomd.xml HTTP/1.1” 403 69 “-” “urlgrabber/3.10 yum/3.4.3”

Katello 3.16

Unable to edit

curl:
403: [(‘PEM routines’, ‘PEM_read_bio’, ‘no start line’)]

@CowsSayMooo thanks for bringing this to our attention. I’ll try and find a solution for you ASAP.

@CowsSayMooo run the following command:
rct cat-cert /etc/pki/entitlement/FILENAME

where FILENAME is each of the non-.pem files in that directory.

1 Like

Hi,
i have the same Problem, described here:

Did you fix the Problem ?

I ended up with another reinstall, worked like a charm afterwards.

In my case I did an Install on antoher server with the same installer command, and the issue persists.

Anyone make any progress on this issue? I have a server rebuild that is encountering the same issue and want to know if any progress has been made troubleshooting this. Any suggestions on how to continue troubleshooting this on my end would be greatly appreciated as well.

Just did a new install (with same setup/version and parameters) as my working server, and this problem encurs again.

Tried creating a new organisation, added a new RHEL subscription (created new subscription allocation and imported new manifest), synced repos, created new CV/Activation keys, but still same error.

This time with RedHat repos only, no other custom repos.

Anyone?

Could this be due to the transition from pulp2 to pulpcore for yum repositories in katello?

I just tested a Foreman/Katello server (2.1/3.16) that is still using pulp2 for yum repos and the manifest imported fine, synced some red hat repos and then verified that I could access the files in said repo via katello.

This is making me strongly suspect that this is a pulp3(pulpcore) problem.

In my case a reinstall without custom certificates fixed it. I Think the installer has Problems with custom certificates. Sadly nobody from the project cares about this thread :frowning:

I don’t use custom certificates, and didnt use during install either. Weird issue.

I think I found the issue here - or at least an issue with using custom certificates.

Katello is still using its self-signed default CA to distribute entitlement certificates. This is expected.

However, pulpcore certguard has the wrong CA configured in its database - it has picked up the Server CA, which should only be used for clients to authenticate the server certificate.

Updating the content of ca_certificate in pulpcore:certguard_rhsmcertguard fixes the issue and allows clients to access the repo.

psql -d pulpcore
pulpcore=# \set content cat /etc/pki/katello/certs/katello-default-ca-stripped.crt``
pulpcore=# update certguard_rhsmcertguard SET ca_certificate = :'content' ;

1 Like

I have just got similar issue after update to Katello 4.0 on my smart proxies.

I use self-signed certificates and even install them from scratch, but it doesn’t help.

Maybe someone has some ideas where is a problem?

I have found solution:

/etc/httpd/conf.d/ssl.conf:
SSLOptions StdEnvVars +ExportCertData

ExportCertData is present everywhere else , but is missing in ssl.conf. That is why there is a problem with passing X-CLIENT-CERT variable to pulpcore-content.

2 Likes

I’ve been fighting this same issue. My original server started at 3.13 and has been upgraded to every release up to 4.0. We also use custom SSL certificates from our internal CA. With the 4.0 upgrade I had this exact issue and couldn’t access the RHEL repos anymore. I rolled back my snapshot to 3.18.2 until I noticed that new CVs were not actually publishing new content. I ended up deploying a new 4.0 server from scratch and again ran into this issue. After redeploying again with the default self signed SSL certificates and then updating to my custom SSL certs I was able to get the main server working. My current problem is with Smart Proxies. I am able to register systems using our custom SSL certs but I can’t access RHEL repos. I had no luck with the ssl.conf change suggested. I’m just finishing up a proxy server rebuild and will continue digging into this issue.

On a side note, the original poster was using 3.16. I have built up servers and smart proxies with 3.17, 3.18, and 4.0 and have verified that this SSL issue is present since all those versions use Pulp3 for content with new builds. My problems all started after I did the pulp 2 to 3 migration. I also tested the main server and proxy using the default self signed certs but couldn’t get the proxy to register because the self signed certs were not trusted.

I have translated this to an issue and will track it down: Bug #32624: Client receives 403 forbidden when fetching RHEL content when using custom certificates - Installer - Foreman