HTTP 403 when using RedHat Repositories

Problem:
After reinstalling our Foreman and reregistering the clients we can not download any RedHat content.
When trying to download content from RedHat repositories from a registered and subscribed client I get [Errno 14] HTTPS Error 403 - Forbidden

Expected outcome:
Working download and install

Foreman and Proxy versions:
2.2.1
Foreman and Proxy plugin versions:
katello 3.17.1

Distribution and version:
CentOS 7.9
Other relevant data:
Already tried:

  • Resubscribe client
  • Refresh manifest
  • Reattach subscription
  • Debug yum
2020-12-14 14:31:31,238 opening local file "/var/cache/yum/x86_64/7Server/rhel-7-server-rpms/repomd4Bx69ltmp.xml" with mode wb
* About to connect() to foreman port 443 (#11)                                                                                                                                                                                                                                                           
*   Trying 10.XXX.XXX.X...
* Connected to foreman (10.XXX.XXX.X) port 443 (#11)
* warning: CURLOPT_CAPATH not a directory (/etc/rhsm/ca/katello-server-ca.pem)
*   CAfile: /etc/rhsm/ca/katello-server-ca.pem
  CApath: /etc/rhsm/ca/katello-server-ca.pem
* NSS: client certificate from file
*       subject: CN=a15ff28fccf44606bd87da09469d6a1c,O=org
*       start date: Feb 01 05:00:00 2019 GMT
*       expire date: Apr 01 03:59:59 2024 GMT                                                                                                                                                                                                                                                                             
*       common name: a15ff28fccf44606bd87da09469d6a1c                                                                                                                                                                                                                                                                     
*       issuer: CN=foreman ,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North Carolina,C=US
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256                                                                                                                                                                                                                                                              
* Server certificate:                                                                                                                                                                                                                                                                                                     
*       subject: CN=foreman
*       start date: Dec 06 09:37:23 2020 GMT
*       expire date: Dec 06 09:37:23 2025 GMT
*       common name: foreman                                                                                                                                                                                                                                                                            
*       issuer: CN=CA-Prod                                                                                                                                                                                                                                                                                          
> GET /pulp/repos/org/Testing/RedHat_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml HTTP/1.1
User-Agent: urlgrabber/3.10 yum/3.4.3
Host: foreman
Accept: */*

* The requested URL returned error: 403 Client certificate is not signed by the stored 'ca_certificate'.
* Closing connection 11

The server log is is not really helpful. Only (known) related output:
Dez 14 14:43:56 foreman pulpcore-content[46212]: 127.0.0.1 [14/Dec/2020:13:43:56 +0000] "GET /pulp/content/uniVersa/Testing/RedHat_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml HTTP/1.1" 403 282 "-" "urlgrabber/3.10 yum/3.4.3"

Can anyone help ?

I investigated the problem since my last post, and when installing a foreman without custom certs, it is working. When installing with custom certs it is broken. When I install the custom certificates after the first install, it is also working. Currently I didn’t manage to get a broken foreman working by doing a cert reset …