403 - Forbidden Error for RHEL Repos with the Clients registered to smart proxy

Support
1

I am experiencing the following issues and I am hoping that someone will be able to help me or point me in the right direction.

Problem:
RHEL 7 Clients registered to the Foreman smart proxy are getting 403 errors on the RHEL repos. Other non-rhel repos like epel, puppet6, etc that exist on the same content view are fine.

Rhel 7 Clients registered to the main foreman servers are fine.

The published content view is synced to the foreman smart proxy and both foreman and smart proxy are installed in CentOS 7.

    subscription-manager list
    Product Name:   Red Hat Enterprise Linux Server
    Product ID:     69
    Version:        7.9
    Arch:           x86_64
    Status:         Subscribed
    Status Details:
    Starts:         12/05/20
    Ends:           12/05/21

    yum repolist
    Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
    epel7                                                                         | 2.3 kB  00:00:00
    foreman_clients_el_7                                            | 2.3 kB  00:00:00
    puppet6_el_7                                                           | 2.0 kB  00:00:00
    
 https://foremansmartproxy.mycompany.com/pulp/content/mycompany/DEV/RHEL_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden
    Trying other mirror.
    To address this issue please refer to the below knowledge base article

   https://access.redhat.com/solutions/69319

   If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.

    epel7/primary                                                                 | 4.9 MB  00:00:00
epel7                                                                                    13585/13585
foreman_clients_el_7/primary                                    | 5.1 kB  00:00:00
foreman_clients_el_7                                                             16/16
puppet6_el_7/primary                                                   | 183 kB  00:00:00
puppet6_el_7                                                                          294/294
https://foreman-smartproxy.mycompany.com/pulp/content/mycompany/DEV/RHEL_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden
Trying other mirror.
repo id                                                         repo name                                         status
epel7                                       epel7                                         13585
foreman_clients_el_7          foreman_clients_el_7                             16
puppet6_el_7                         puppet6_el_7                                    294
rhel-7-server-rpms/7Server/x86_64                               Red Hat Enterprise Linux 7 Server (RPMs)              0
repolist: 13895

Expected outcome:

Clients should be able to fetch RHEL 7 rpms from smart proxy.

Foreman and Proxy versions:

Foreman 2.4, Katello 4.0.

Foreman and Proxy plugin versions:

Foreman 2.4, Katello 4.0.

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:
Content Sync Smart proxy is synchronized Last sync: 2021-05-04 10:50:41 -0500

3

Sample logs:

May 5 10:37:06 smart-proxy-server pulpcore-content: [05/May/2021:14:37:06 +0000] "GET /pulp/content/ORG/DEV/RHEL_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml HTTP/1.1" 403 292 "-" "urlgrabber/3.10 yum/3.4.3 "

Got the following error while accessing the published content in the browser:

403: A client certificate was not received via the X-CLIENT-CERT header.

4

You’ll have to use the client certificate to access the repo. To check if it works, check the repo file /etc/yum.repos.d/redhat.repo. Look for the repository with the problem, i.e. rhel-7-server-rpms. It looks something like this:

[rhel-7-server-rpms]
...
sslclientcert = /etc/pki/entitlement/5408444497409024036.pem
baseurl = https://foreman-smartproxy.example.com/pulp/content/mycompany/DEV/RHEL_7/content/dist/rhel/server/7/$releasever/$basearch/os
...
sslclientkey = /etc/pki/entitlement/5408444497409024036-key.pem
...

Use curl to check:

# curl --cert /etc/pki/entitlement/5408444497409024036.pem --key /etc/pki/entitlement/5408444497409024036-key.pem https://foreman-smartproxy.example.com/pulp/content/mycompany/DEV/RHEL_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml

If you still get the 403 then it seems it has an incorrect entitlement certificate. Maybe a subscription-manager refresh can help then.

1 Like
5

@gvde

Thanks for your quick response.

I am able to curl from the client to main foreman server just fine as below:

    [root@rhel7-client entitlement]# curl --cert /etc/pki/entitlement/1373568910360318289.pem --key /etc/pki/entitlement/1373568910360318289-key.pem https://mainForemanServer.example.com/pulp/content/ORG/DEV/RHEL_7/content/dist/rhel/server/7/7Server
    /x86_64/os/repodata/repomd.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
      <revision>1619789679</revision>
..etc

But if I try to fetch repomd.xml or any other content from foreman smart proxy using the same entitlement, that’s when I get the 403 as follows:

[root@rhel7-client  entitlement]# curl --cert /etc/pki/entitlement/1373568910360318289.pem --key /etc/pki/entitlement/1373568910360318289-key.pem https://ForemanSmartProxy.example.com/pulp/content/ORG/DEV/RHEL_7/content/dist/rhel/server/7/7Server
/x86_64/os/repodata/repomd.xml
403: A client certificate was not received via the `X-CLIENT-CERT` header.[root@rhel7-client entitlement]#

Also did subscription-manager refresh with no luck.

6

I have exactly the same issue on Katello 4.0 with Foreman 2.4.

With master foreman curl gives me content, but with smart-proxies I get:

403: A client certificate was not received via the X-CLIENT-CERT

7

Is your’s only for RHEL repos also? Mine works for others but I get SSL errors on RHEL repos as described.

Thanks,

8

I’ve created Bug #32622: RHEL content on a content proxy returns a 403 to clients - Installer - Foreman to track this issue and am starting to investigate.

9

Awesome!! Thank you. :pray:

10

I have the same issue on my new Katello 4.0 deployment. I also confirmed the same issue is present on 3.17 and 3.18. In my situation I am using custom SSL certificates on Katello and Smart Proxies. Also the only way the Katello server would work correctly is to deploy using the default self signed SSL certificates then update to my custom certs. I did update my SSL certificates to include the required extensions stated in the install document. My previous server with pulp 2 worked just fine without them until I migrated content to pulp 3. Subscription manager and non-RHEL repositories work fine through the smart proxy. The only issue left is this 403 error on RHEL repos.

11

If I update /etc/httpd/conf.d/ssl.conf as stated in: [Errno 14] HTTPS Error 403 - Forbidden (RedHat repositories only) - #16 by radekb

I get “403: Client certificate is not signed by the stored ‘ca_certificate’.”

12

I have opened a proposed permanent fix for this issue that we will have to backport. In the meantime you can locally patch your content proxy:

Edit /etc/httpd/conf.d/10-pulpcore-https.conf and add the line SSLOptions +StdEnvVars +ExportCertData as shown below:

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/pki/katello/certs/katello-apache.crt"
  SSLCertificateKeyFile   "/etc/pki/katello/private/katello-apache.key"
  SSLCertificateChainFile "/etc/pki/katello/certs/katello-server-ca.crt"
  SSLVerifyClient         optional
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt"
  SSLOptions +StdEnvVars +ExportCertData

Then restart httpd: systemctl restart httpd

NOTE: Any re-run of the installer will wipe this change out until we have backported a fix.

A slightly more installer proof method is to edit /etc/foreman-installer/custom-hiera.yaml and add:

pulpcore::apache::https_vhost_options:
  ssl_options: '+StdEnvVars +ExportCertData'

You’ll just want to clear this out after the backport, and the installer should help with that by throwing an error once the fix is officially available.

2 Likes
13

Great. This worked for me.

14

It worked for me but I also had to update the CA cert in the pulpcore database.

su - postgres
psql -d pulpcore
\set content `cat /etc/pki/katello/certs/katello-default-ca-stripped.crt`
update certguard_rhsmcertguard SET ca_certificate = :‘content’ ;

15

Hi

I have kinda the same problem and nothing mentioned seems to help.
Problem is when i trying to start RHEL8 installation in kickstart i have AppStream repo pointed like this.
Install fails as it can not access AppStream repo.

> url --url http://katello.server.com/pulp/repos/BB/Library/content/dist/rhel8/8.9/x86_64/baseos/kickstart
> repo --name AppStream --baseurl http://katello.server.com/pulp/repos/BB/Library/content/dist/rhel8/8.9/x86_64/appstream/os

But katello does not allow to share AppStream repo via http as it requires certificates to access RHEL repos and it can only be seen with client certificate which can not be provided in kickstart install.
Is there are any workaround for this ?
I need to be able to access RHEL repos without ssl certificate.
Thanks

16

You need to sync the Appstream kickstart repos, too, then you can use repo --name AppStream --baseurl http://katello.server.com/pulp/repos/BB/Library/content/dist/rhel8/8.9/x86_64/appstream/kickstart

17

Thank you a lot. Seems like i sorted out. Anyway i needed to resync everything as you said and repos in kickstart should look like this, then it will work.

install

url --url http://katello.server.com/pulp/repos/BB/Library/content/dist/rhel8/8.9/x86_64/baseos/kickstart
repo --name AppStream --baseurl https://katello.server.com/pulp/content/BB/Library/content/dist/rhel8/8.9/x86_64/appstream/kickstart