Katello gives json error and 401 Unauthorized when attempting to register using subscription-manager

reesethegeek · May 2, 2020, 4:39pm

Problem:
While attempting to subscription-manager register a Centos7 machine to our Katello, we are presented with a json error and it fails and a 401 error because of the json error. We also see "Unauthorized: Invalid Credentials for request. This is stopping our much needed migration off the Spacewalk platform.

We have all the repos synced successfully
We have a Lifecycle Environment Created
We have content views created for Centos 7
We have an Activation key that is linked to that Environment/content view

Expected outcome:
We need a successful register of the centos7 host to the katello platform so that it can successfully use the Repositories via subscription that are marked for centos7 usage and properly install and update pages to keep our security posture up to date.

Foreman and Proxy versions:
foreman.noarch 2.0.0-1.el7
foreman-cli.noarch 2.0.0-1.el7
foreman-debug.noarch 2.0.0-1.el7
foreman-dynflow-sidekiq.noarch 2.0.0-1.el7
foreman-ec2.noarch 2.0.0-1.el7
foreman-installer.noarch 1:2.0.0-1.el7
foreman-installer-katello.noarch 1:2.0.0-1.el7
foreman-postgresql.noarch 2.0.0-1.el7
foreman-proxy.noarch 2.0.0-1.el7
foreman-release.noarch 2.0.0-1.el7
foreman-release-scl.noarch 7-2.el7

foreman.here.com-apache.noarch 1.0-1 installed
foreman.here.com-foreman-client.noarch
foreman.here.com-foreman-proxy.noarch
foreman.here.com-foreman-proxy-client.noarch
foreman.here.com-puppet-client.noarch
foreman.here.com-qpid-broker.noarch
foreman.here.com-qpid-client-cert.noarch
foreman.here.com-qpid-router-client.noarch
foreman.here.com-qpid-router-server.noarch
foreman.here.com-tomcat.noarch 1.0-1 installed

Foreman and Proxy plugin versions:
candlepin.noarch 3.1.7-1.el7
candlepin-selinux.noarch 3.1.7-1.el7
foreman-installer-katello.noarch 1:2.0.0-1.el7
katello.noarch 3.15.0.1-1.el7
katello-certs-tools.noarch 2.6.0-1.el7
katello-client-bootstrap.noarch 1.7.4-1.el7

Distribution and version:
Katello version 3.15.0.1-1.el7
Foreman version 2.0.0-1.el7

Other relevant data:

Foreman is installed on a Centos 7 host - no errors during the installation - used self signed certificates and properly validated with the certificate check
Foreman is fronted by an Amazon Web Services Classic Load Balancers with a HTTPS cert on the load balancer
- has the following listeners on the load balancer
  80,443,5646, 5647, 8000, 8140, 9090, 7, 53, 67, 69, 5000
The machine is a m5a.xlarge - no issues on resources

LOG
` 2020-05-02 15:59:20,220 [INFO] subscription-manager:6873:MainThread @connection.py:905 - Connection built: host=subscription.rhsm.redhat.com port=443 handler=/subscription auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 15:59:21,005 [INFO] subscription-manager:6905:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 15:59:21,448 [INFO] subscription-manager:6917:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:04:01,801 [INFO] yum:8214:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:04:01,802 [INFO] yum:8214:MainThread @entcertlib.py:131 - certs updated:

Total updates: 0

Found (local) serial# []

Expected (UEP) serial# []

Added (new)

<NONE>

Deleted (rogue):

<NONE>

2020-05-02 16:08:27,101 [INFO] yum:9656:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:08:27,102 [INFO] yum:9656:MainThread @entcertlib.py:131 - certs updated:

Total updates: 0

Found (local) serial# []

Expected (UEP) serial# []

Added (new)

<NONE>

Deleted (rogue):

<NONE>

2020-05-02 16:09:53,666 [INFO] yum:10197:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:09:53,666 [INFO] yum:10197:MainThread @entcertlib.py:131 - certs updated:

Total updates: 0

Found (local) serial# []

Expected (UEP) serial# []

Added (new)

<NONE>

Deleted (rogue):

<NONE>

2020-05-02 16:11:07,710 [WARNING] yum:10197:MainThread @logutil.py:142 - logging already initialized

2020-05-02 16:13:06,657 [INFO] yum:11633:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:13:06,658 [INFO] yum:11633:MainThread @entcertlib.py:131 - certs updated:

Total updates: 0

Found (local) serial# []

Expected (UEP) serial# []

Added (new)

<NONE>

Deleted (rogue):

<NONE>

2020-05-02 16:13:21,371 [WARNING] yum:11633:MainThread @logutil.py:142 - logging already initialized

2020-05-02 16:13:40,826 [INFO] yum:11953:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:13:40,827 [INFO] yum:11953:MainThread @entcertlib.py:131 - certs updated:

Total updates: 0

Found (local) serial# []

Expected (UEP) serial# []

Added (new)

<NONE>

Deleted (rogue):

<NONE>

2020-05-02 16:17:28,025 [INFO] subscription-manager:13181:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:17:28,044 [INFO] subscription-manager:13181:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False

2020-05-02 16:17:28,045 [INFO] subscription-manager:13181:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=none

2020-05-02 16:17:28,046 [INFO] subscription-manager:13181:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=none

2020-05-02 16:17:31,098 [INFO] subscription-manager:13181:MainThread @managerlib.py:72 - Consumer created: foreman123.here.com (cdab5789-7b1c-4d6e-86ae-ff90560a07c3)

2020-05-02 16:17:31,100 [INFO] subscription-manager:13181:MainThread @connection.py:905 - Connection built: host=foreman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2020-05-02 16:17:31,180 [ERROR] subscription-manager:13181:MainThread @connection.py:643 - Response: 401
2020-05-02 16:17:31,180 [ERROR] subscription-manager:13181:MainThread @connection.py:644 - JSON parsing error: Expecting ':' delimiter: line 1 column 9 (char 8)
**2020-05-02 16:17:31,180 [ERROR] subscription-manager:13181:MainThread @managercli.py:215 - Error during registration: Server error attempting a GET to /rhsm/consumers/cdab5789-7b1c-4d6e-86ae-ff90560a07c3 returned status 401**
**Unauthorized: Invalid credentials for request.**
**2020-05-02 16:17:31,180 [ERROR] subscription-manager:13181:MainThread @managercli.py:216 - Server error attempting a GET to /rhsm/consumers/cdab5789-7b1c-4d6e-86ae-ff90560a07c3 returned status 401**
Unauthorized: Invalid credentials for request.

Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 1362, in _do_command
type=self.options.consumertype
File "/usr/lib64/python2.7/site-packages/rhsmlib/services/register.py", line 104, in register
store.sync()
File "/usr/lib/python2.7/site-packages/syspurpose/files.py", line 281, in sync
remote_contents = self.get_remote_contents()
File "/usr/lib/python2.7/site-packages/syspurpose/files.py", line 340, in get_remote_contents
consumer = self.uep.getConsumer(self.consumer_uuid)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1195, in getConsumer
return self.conn.request_get(method)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 726, in request_get
return self._request("GET", method, headers=headers)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 752, in _request
info=info, headers=headers)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 627, in _request
self.validateResponse(result, request_type, handler)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 699, in validateResponse
handler=handler)
UnauthorizedException: Server error attempting a GET to /rhsm/consumers/cdab5789-7b1c-4d6e-86ae-ff90560a07c3 returned status 401
Unauthorized: Invalid credentials for request.`

Jonathon_Turel · May 3, 2020, 10:16pm

Hi @reesethegeek,

I noticed in your logs (thanks for providing the snippet!) that your client host is trying to contact subscription.rhsm.redhat.com - that’s incorrect if you are trying to register it to your Foreman+Katello instance.

The corrective action here is to install Katello’s consumer RPM onto the client system and then register. The consumer RPM will ensure that subscription-manager is talking to the right host (your Foreman server in this case). It’s available at http://yourforemaninstance.example.com/pub/katello-ca-consumer-latest.noarch.rpm and it will install the necessary SSL certs and configure subscription-manager in the correct way.

reesethegeek · May 4, 2020, 1:05pm

Hello @Jonathon_Turel - That was the initial installation of subscription manager. As you can see any subsequent log entries show that the katello consumer rpm was indeed downloaded and installed from our forman server.

Jonathon_Turel · May 4, 2020, 9:10pm

Ah, I see. On the Foreman server do you see corresponding 401 errors in /var/log/foreman/production.log? I have a feeling that your load balancer is returning the 401 and the SSL termination really needs to be done by the Foreman server rather than the load balancer.

reesethegeek · May 5, 2020, 12:28am

Hey @Jonathon_Turel
We changed our dns entry to a host record aname (removed the load balancer)

Now we get this

[root@ip-10-103-136-120 yum.repos.d]# yum repolist
Loaded plugins: enabled_repos_upload, package_upload, presto, product-id, search-disabled-repos, security,
              : subscription-manager
https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-CentosPlus/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 503"
Trying other mirror.
https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-CentosPlus/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 503"
Trying other mirror.
https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-Extras/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 503"
Trying other mirror.
https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-FastTrack/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 503"
Trying other mirror.
https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-OS/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 503"
Trying other mirror.
https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-Updates/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 503"
Trying other mirror.
repo id                                  repo name                                                       status
CHE_Centos6_Centos6_10-CentosPlus        Centos6.10-CentosPlus                                                0
CHE_Centos6_Centos6_10-Extras            Centos6.10-Extras                                                    0
CHE_Centos6_Centos6_10-FastTrack         Centos6.10-FastTrack                                                 0
CHE_Centos6_Centos6_10-OS                Centos6.10-OS                                                        0
CHE_Centos6_Centos6_10-Updates           Centos6.10-Updates                                                   0
base                                     CentOS-6 - Base                                                  7,539
epel                                     Extra Packages for Enterprise Linux 7 - x86_64                  13,250
epel-subscription-manager                Tools and libraries for Red Hat subscription management.            23
extras                                   CentOS-6 - Extras                                                   70
foreman-client                           Foreman client 2.0                                                   6
group_qpid-qpid                          Copr repo for qpid owned by @qpid                                   28
newrelic-infra                           New Relic Infrastructure                                           442
pulp                                     Pulp Community Release                                              21
splunk                                   HERE Splunk UF                                                      3
subscription-manager                     Subscription manager repository from Candlepin                      10
updates This text will be hidden                                 CentOS-6 - Updates                                               2,281


The log looks like this:

    2020-05-04 03:44:01,984 [INFO] rhsmd @rhsmd:302 - rhsmd started
2020-05-04 03:44:01,984 [INFO] rhsmd @rhsmd:333 - logging subscription status to syslog
2020-05-04 03:44:01,985 [DEBUG] rhsmd @identity.py:131 - Loading consumer info from identity certificates.
2020-05-04 03:44:01,990 [DEBUG] rhsmd @profile.py:97 - Loading current RPM profile.
2020-05-04 03:44:02,067 [INFO] rhsmd @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 03:44:02,068 [INFO] rhsmd @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 03:44:02,068 [INFO] rhsmd @cache.py:138 - Checking current system info against cache: /var/lib/rhsm/cache/installed_products.json
2020-05-04 03:44:02,068 [INFO] rhsmd @cache.py:155 - No changes.
2020-05-04 03:44:02,069 [DEBUG] rhsmd @certdirectory.py:216 - Installed product IDs: []
2020-05-04 03:44:02,070 [DEBUG] rhsmd @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 03:44:02,070 [DEBUG] rhsmd @connection.py:466 - Making request: GET /rhsm/consumers/f2808918-3e5c-43e4-bf72-7145a7462903/compliance
2020-05-04 03:44:02,317 [DEBUG] rhsmd @connection.py:489 - Response: status=403
2020-05-04 17:03:05,312 [DEBUG] yum @identity.py:131 - Loading consumer info from identity certificates.
2020-05-04 17:03:05,317 [DEBUG] yum @profile.py:97 - Loading current RPM profile.
2020-05-04 17:03:05,397 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 17:03:05,397 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 17:03:05,399 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 17:03:05,399 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 17:03:05,400 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 17:03:05,401 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/
2020-05-04 17:03:05,560 [DEBUG] yum @connection.py:489 - Response: status=200
2020-05-04 17:03:05,561 [DEBUG] yum @connection.py:706 - Server supports the following resources:
2020-05-04 17:03:05,561 [DEBUG] yum @connection.py:707 - {'available_releases': '/rhsm/consumers/:id/available_releases', 'status': '/rhsm/status', 'deb_package_profile': '/rhsm/systems/:id/deb_package_profile', 'guestids': '/rhsm/consumers/:id/guestids', 'content_overrides': '/rhsm/consumers/:id/content_overrides', 'environments': '/rhsm/owners/:organization_id/environments', 'hypervisors': '/rhsm/hypervisors', 'owner': '/rhsm/consumers/:id/owner', 'certificates': '/rhsm/consumers/:consumer_id/certificates', 'servicelevels': '/rhsm/owners/:organization_id/servicelevels', 'serials': '/rhsm/consumers/:id/certificates/serials', 'deleted_consumers': '/rhsm/deleted_consumers', 'consumers': '/rhsm/environments/:environment_id/consumers', 'accessible_content': '/rhsm/consumers/:id/accessible_content', 'entitlements': '/rhsm/entitlements', 'profile': '/rhsm/consumers/:id/profile', 'dry-run': '/rhsm/consumers/:id/entitlements/dry-run', 'subscriptions': '/rhsm/subscriptions', 'checkin': '/rhsm/consumers/:id/checkin', 'deletionrecord': '/rhsm/consumers/:id/deletionrecord', 'release': '/rhsm/consumers/:id/release', ':poolId': '/rhsm/consumers/:id/entitlements/pool/:poolId', ':jobId': '/rhsm/jobs/:jobId', 'packages': '/rhsm/consumers/:id/packages', 'owners': '/rhsm/users/:login/owners', 'compliance': '/rhsm/consumers/:id/compliance', ':owner': '/rhsm/hypervisors/:owner', 'profiles': '/rhsm/consumers/:id/profiles', 'enabled_repos': '/rhsm/systems/:id/enabled_repos', 'pools': '/rhsm/pools', 'purpose_compliance': '/rhsm/consumers/:id/purpose_compliance', 'tracer': '/rhsm/consumers/:id/tracer'}
2020-05-04 17:03:05,562 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 17:03:05,563 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/consumers/f2808918-3e5c-43e4-bf72-7145a7462903/content_overrides
2020-05-04 17:03:05,718 [DEBUG] yum @connection.py:489 - Response: status=404
2020-05-04 17:03:05,719 [DEBUG] yum @__init__.py:85 - Searching for content of type: yum
2020-05-04 17:03:05,720 [DEBUG] yum @cache.py:110 - Wrote cache: /var/lib/rhsm/cache/written_overrides.json
2020-05-04 17:03:05,720 [INFO] yum @repolib.py:270 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-05-04 23:15:16,179 [DEBUG] yum @identity.py:131 - Loading consumer info from identity certificates.
2020-05-04 23:15:16,184 [DEBUG] yum @profile.py:97 - Loading current RPM profile.
2020-05-04 23:15:16,263 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 23:15:16,264 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 23:15:16,265 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 23:15:16,266 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 23:15:16,267 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 23:15:16,267 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/
2020-05-04 23:15:16,461 [DEBUG] yum @connection.py:489 - Response: status=200
2020-05-04 23:15:16,462 [DEBUG] yum @connection.py:706 - Server supports the following resources:
2020-05-04 23:15:16,462 [DEBUG] yum @connection.py:707 - {'available_releases': '/rhsm/consumers/:id/available_releases', 'status': '/rhsm/status', 'deb_package_profile': '/rhsm/systems/:id/deb_package_profile', 'guestids': '/rhsm/consumers/:id/guestids', 'content_overrides': '/rhsm/consumers/:id/content_overrides', 'environments': '/rhsm/owners/:organization_id/environments', 'hypervisors': '/rhsm/hypervisors', 'owner': '/rhsm/consumers/:id/owner', 'certificates': '/rhsm/consumers/:consumer_id/certificates', 'servicelevels': '/rhsm/owners/:organization_id/servicelevels', 'serials': '/rhsm/consumers/:id/certificates/serials', 'deleted_consumers': '/rhsm/deleted_consumers', 'consumers': '/rhsm/environments/:environment_id/consumers', 'accessible_content': '/rhsm/consumers/:id/accessible_content', 'entitlements': '/rhsm/entitlements', 'profile': '/rhsm/consumers/:id/profile', 'dry-run': '/rhsm/consumers/:id/entitlements/dry-run', 'subscriptions': '/rhsm/subscriptions', 'checkin': '/rhsm/consumers/:id/checkin', 'deletionrecord': '/rhsm/consumers/:id/deletionrecord', 'release': '/rhsm/consumers/:id/release', ':poolId': '/rhsm/consumers/:id/entitlements/pool/:poolId', ':jobId': '/rhsm/jobs/:jobId', 'packages': '/rhsm/consumers/:id/packages', 'owners': '/rhsm/users/:login/owners', 'compliance': '/rhsm/consumers/:id/compliance', ':owner': '/rhsm/hypervisors/:owner', 'profiles': '/rhsm/consumers/:id/profiles', 'enabled_repos': '/rhsm/systems/:id/enabled_repos', 'pools': '/rhsm/pools', 'purpose_compliance': '/rhsm/consumers/:id/purpose_compliance', 'tracer': '/rhsm/consumers/:id/tracer'}
2020-05-04 23:15:16,463 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 23:15:16,464 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/consumers/f2808918-3e5c-43e4-bf72-7145a7462903/content_overrides
2020-05-04 23:15:16,682 [DEBUG] yum @connection.py:489 - Response: status=404
2020-05-04 23:15:16,683 [DEBUG] yum @__init__.py:85 - Searching for content of type: yum
2020-05-04 23:15:16,684 [DEBUG] yum @cache.py:110 - Wrote cache: /var/lib/rhsm/cache/written_overrides.json
2020-05-04 23:15:16,684 [INFO] yum @repolib.py:270 - repos updated: Repo updates
2020-05-04 23:15:24,804 [DEBUG] yum @plugins.py:569 - loaded plugin modules: []
2020-05-04 23:15:24,805 [DEBUG] yum @plugins.py:570 - loaded plugins: {}
2020-05-04 23:15:24,805 [WARNING] yum @productid.py:736 - Error loading productid metadata for base.
2020-05-04 23:15:24,805 [WARNING] yum @productid.py:736 - Error loading productid metadata for epel.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for epel-subscription-manager.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for extras.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for foreman-client.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for group_qpid-qpid.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for newrelic-infra.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for pulp.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for splunk.
2020-05-04 23:15:24,806 [WARNING] yum @productid.py:736 - Error loading productid metadata for subscription-manager.
2020-05-04 23:15:24,807 [WARNING] yum @productid.py:736 - Error loading productid metadata for updates.
2020-05-04 23:15:25,769 [DEBUG] yum @productid.py:408 - Checking for product id certs to install or update.
2020-05-04 23:15:25,770 [DEBUG] yum @productid.py:546 - about to run post_product_id_install
2020-05-04 23:15:25,770 [DEBUG] yum @productid.py:557 - about to run post_product_id_update
2020-05-04 23:15:42,378 [DEBUG] yum @identity.py:131 - Loading consumer info from identity certificates.
2020-05-04 23:15:42,383 [DEBUG] yum @profile.py:97 - Loading current RPM profile.
2020-05-04 23:15:42,460 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 23:15:42,460 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 23:15:42,462 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 23:15:42,462 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 23:15:42,463 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 23:15:42,464 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/
2020-05-04 23:15:42,611 [DEBUG] yum @connection.py:489 - Response: status=200
2020-05-04 23:15:42,612 [DEBUG] yum @connection.py:706 - Server supports the following resources:
2020-05-04 23:15:42,613 [DEBUG] yum @connection.py:707 - {'available_releases': '/rhsm/consumers/:id/available_releases', 'status': '/rhsm/status', 'deb_package_profile': '/rhsm/systems/:id/deb_package_profile', 'guestids': '/rhsm/consumers/:id/guestids', 'content_overrides': '/rhsm/consumers/:id/content_overrides', 'environments': '/rhsm/owners/:organization_id/environments', 'hypervisors': '/rhsm/hypervisors', 'owner': '/rhsm/consumers/:id/owner', 'certificates': '/rhsm/consumers/:consumer_id/certificates', 'servicelevels': '/rhsm/owners/:organization_id/servicelevels', 'serials': '/rhsm/consumers/:id/certificates/serials', 'deleted_consumers': '/rhsm/deleted_consumers', 'consumers': '/rhsm/environments/:environment_id/consumers', 'accessible_content': '/rhsm/consumers/:id/accessible_content', 'entitlements': '/rhsm/entitlements', 'profile': '/rhsm/consumers/:id/profile', 'dry-run': '/rhsm/consumers/:id/entitlements/dry-run', 'subscriptions': '/rhsm/subscriptions', 'checkin': '/rhsm/consumers/:id/checkin', 'deletionrecord': '/rhsm/consumers/:id/deletionrecord', 'release': '/rhsm/consumers/:id/release', ':poolId': '/rhsm/consumers/:id/entitlements/pool/:poolId', ':jobId': '/rhsm/jobs/:jobId', 'packages': '/rhsm/consumers/:id/packages', 'owners': '/rhsm/users/:login/owners', 'compliance': '/rhsm/consumers/:id/compliance', ':owner': '/rhsm/hypervisors/:owner', 'profiles': '/rhsm/consumers/:id/profiles', 'enabled_repos': '/rhsm/systems/:id/enabled_repos', 'pools': '/rhsm/pools', 'purpose_compliance': '/rhsm/consumers/:id/purpose_compliance', 'tracer': '/rhsm/consumers/:id/tracer'}
2020-05-04 23:15:42,614 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 23:15:42,615 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/consumers/f2808918-3e5c-43e4-bf72-7145a7462903/content_overrides
2020-05-04 23:15:42,757 [DEBUG] yum @connection.py:489 - Response: status=404
2020-05-04 23:15:42,758 [DEBUG] yum @__init__.py:85 - Searching for content of type: yum
2020-05-04 23:15:42,759 [DEBUG] yum @cache.py:110 - Wrote cache: /var/lib/rhsm/cache/written_overrides.json
2020-05-04 23:15:42,759 [INFO] yum @repolib.py:270 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-05-04 23:15:52,828 [DEBUG] yum @identity.py:131 - Loading consumer info from identity certificates.
2020-05-04 23:15:52,833 [DEBUG] yum @profile.py:97 - Loading current RPM profile.
2020-05-04 23:15:52,907 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 23:15:52,907 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 23:15:52,908 [INFO] yum @connection.py:679 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2020-05-04 23:15:52,908 [INFO] yum @connection.py:690 - Connection Built: host: forman.here.com, port: 443, handler: /rhsm
2020-05-04 23:15:52,909 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 23:15:52,910 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/
2020-05-04 23:15:53,053 [DEBUG] yum @connection.py:489 - Response: status=200
2020-05-04 23:15:53,053 [DEBUG] yum @connection.py:706 - Server supports the following resources:
2020-05-04 23:15:53,054 [DEBUG] yum @connection.py:707 - {'available_releases': '/rhsm/consumers/:id/available_releases', 'status': '/rhsm/status', 'deb_package_profile': '/rhsm/systems/:id/deb_package_profile', 'guestids': '/rhsm/consumers/:id/guestids', 'content_overrides': '/rhsm/consumers/:id/content_overrides', 'environments': '/rhsm/owners/:organization_id/environments', 'hypervisors': '/rhsm/hypervisors', 'owner': '/rhsm/consumers/:id/owner', 'certificates': '/rhsm/consumers/:consumer_id/certificates', 'servicelevels': '/rhsm/owners/:organization_id/servicelevels', 'serials': '/rhsm/consumers/:id/certificates/serials', 'deleted_consumers': '/rhsm/deleted_consumers', 'consumers': '/rhsm/environments/:environment_id/consumers', 'accessible_content': '/rhsm/consumers/:id/accessible_content', 'entitlements': '/rhsm/entitlements', 'profile': '/rhsm/consumers/:id/profile', 'dry-run': '/rhsm/consumers/:id/entitlements/dry-run', 'subscriptions': '/rhsm/subscriptions', 'checkin': '/rhsm/consumers/:id/checkin', 'deletionrecord': '/rhsm/consumers/:id/deletionrecord', 'release': '/rhsm/consumers/:id/release', ':poolId': '/rhsm/consumers/:id/entitlements/pool/:poolId', ':jobId': '/rhsm/jobs/:jobId', 'packages': '/rhsm/consumers/:id/packages', 'owners': '/rhsm/users/:login/owners', 'compliance': '/rhsm/consumers/:id/compliance', ':owner': '/rhsm/hypervisors/:owner', 'profiles': '/rhsm/consumers/:id/profiles', 'enabled_repos': '/rhsm/systems/:id/enabled_repos', 'pools': '/rhsm/pools', 'purpose_compliance': '/rhsm/consumers/:id/purpose_compliance', 'tracer': '/rhsm/consumers/:id/tracer'}
2020-05-04 23:15:53,055 [DEBUG] yum @connection.py:420 - Loaded CA certificates from /etc/rhsm/ca/: katello-server-ca.pem, redhat-uep.pem, katello-default-ca.pem, candlepin-stage.pem
2020-05-04 23:15:53,055 [DEBUG] yum @connection.py:466 - Making request: GET /rhsm/consumers/f2808918-3e5c-43e4-bf72-7145a7462903/content_overrides
2020-05-04 23:15:53,199 [DEBUG] yum @connection.py:489 - Response: status=404
2020-05-04 23:15:53,201 [DEBUG] yum @__init__.py:85 - Searching for content of type: yum
2020-05-04 23:15:53,201 [DEBUG] yum @cache.py:110 - Wrote cache: /var/lib/rhsm/cache/written_overrides.json
2020-05-04 23:15:53,202 [INFO] yum @repolib.py:270 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-05-04 23:15:56,797 [DEBUG] yum @plugins.py:569 - loaded plugin modules: []
2020-05-04 23:15:56,798 [DEBUG] yum @plugins.py:570 - loaded plugins: {}
2020-05-04 23:15:56,798 [WARNING] yum @productid.py:736 - Error loading productid metadata for base.
2020-05-04 23:15:56,798 [WARNING] yum @productid.py:736 - Error loading productid metadata for epel.
2020-05-04 23:15:56,798 [WARNING] yum @productid.py:736 - Error loading productid metadata for epel-subscription-manager.
2020-05-04 23:15:56,798 [WARNING] yum @productid.py:736 - Error loading productid metadata for extras.
2020-05-04 23:15:56,798 [WARNING] yum @productid.py:736 - Error loading productid metadata for foreman-client.
2020-05-04 23:15:56,798 [WARNING] yum @productid.py:736 - Error loading productid metadata for group_qpid-qpid.
2020-05-04 23:15:56,798 [WARNING] yum @productid.py:736 - Error loading productid metadata for newrelic-infra.
2020-05-04 23:15:56,799 [WARNING] yum @productid.py:736 - Error loading productid metadata for pulp.
2020-05-04 23:15:56,799 [WARNING] yum @productid.py:736 - Error loading productid metadata for splunk.
2020-05-04 23:15:56,799 [WARNING] yum @productid.py:736 - Error loading productid metadata for subscription-manager.
2020-05-04 23:15:56,799 [WARNING] yum @productid.py:736 - Error loading productid metadata for updates.
2020-05-04 23:15:57,780 [DEBUG] yum @productid.py:408 - Checking for product id certs to install or update.
2020-05-04 23:15:57,781 [DEBUG] yum @productid.py:546 - about to run post_product_id_install
2020-05-04 23:15:57,786 [DEBUG] yum @productid.py:557 - about to run post_product_id_update
2020-05-04 23:16:31,035 [INFO] yum:1502 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:16:31,036 [INFO] yum:1502 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:16:31,318 [ERROR] yum:1502 @cache.py:209 - Unit with ID "f2808918-3e5c-43e4-bf72-7145a7462903" could not be found.
Traceback (most recent call last):
  File "/usr/share/rhsm/subscription_manager/cache.py", line 197, in load_status
    self._sync_with_server(uep, uuid)
  File "/usr/share/rhsm/subscription_manager/cache.py", line 331, in _sync_with_server
    self.server_status = uep.getContentOverrides(consumer_uuid)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 1342, in getContentOverrides
    return self.conn.request_get(method)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 681, in request_get
    return self._request("GET", method)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 598, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 648, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Unit with ID "f2808918-3e5c-43e4-bf72-7145a7462903" could not be found.
2020-05-04 23:16:31,320 [INFO] yum:1502 @repolib.py:303 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-05-04 23:16:46,658 [INFO] yum:1504 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:16:46,659 [INFO] yum:1504 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:16:46,935 [ERROR] yum:1504 @cache.py:209 - Unit with ID "f2808918-3e5c-43e4-bf72-7145a7462903" could not be found.
Traceback (most recent call last):
  File "/usr/share/rhsm/subscription_manager/cache.py", line 197, in load_status
    self._sync_with_server(uep, uuid)
  File "/usr/share/rhsm/subscription_manager/cache.py", line 331, in _sync_with_server
    self.server_status = uep.getContentOverrides(consumer_uuid)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 1342, in getContentOverrides
    return self.conn.request_get(method)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 681, in request_get
    return self._request("GET", method)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 598, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 648, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Unit with ID "f2808918-3e5c-43e4-bf72-7145a7462903" could not be found.
2020-05-04 23:16:46,937 [INFO] yum:1504 @repolib.py:303 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-05-04 23:16:52,913 [INFO] subscription-manager:1518 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el6', 'subscription-manager': '1.17.6-1.el6'}
2020-05-04 23:16:53,328 [INFO] subscription-manager:1520 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el6', 'subscription-manager': '1.17.6-1.el6'}
2020-05-04 23:17:04,126 [INFO] yum:1539 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:17:04,127 [INFO] yum:1539 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:17:04,403 [ERROR] yum:1539 @cache.py:209 - Unit with ID "f2808918-3e5c-43e4-bf72-7145a7462903" could not be found.
Traceback (most recent call last):
  File "/usr/share/rhsm/subscription_manager/cache.py", line 197, in load_status
    self._sync_with_server(uep, uuid)
  File "/usr/share/rhsm/subscription_manager/cache.py", line 331, in _sync_with_server
    self.server_status = uep.getContentOverrides(consumer_uuid)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 1342, in getContentOverrides
    return self.conn.request_get(method)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 681, in request_get
    return self._request("GET", method)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 598, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 648, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Unit with ID "f2808918-3e5c-43e4-bf72-7145a7462903" could not be found.
2020-05-04 23:17:04,405 [INFO] yum:1539 @repolib.py:303 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-05-04 23:18:26,320 [INFO] subscription-manager:1617 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el6', 'subscription-manager': '1.17.6-1.el6'}
2020-05-04 23:18:26,320 [INFO] subscription-manager:1617 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:18:26,320 [INFO] subscription-manager:1617 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=none
2020-05-04 23:18:26,321 [INFO] subscription-manager:1617 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el6', 'subscription-manager': '1.17.6-1.el6'}
2020-05-04 23:18:26,321 [INFO] subscription-manager:1617 @managercli.py:359 - Consumer Identity name=f2808918-3e5c-43e4-bf72-7145a7462903, ip-10-103-136-120 uuid=f2808918-3e5c-43e4-bf72-7145a7462903
2020-05-04 23:18:33,706 [INFO] subscription-manager:1618 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el6', 'subscription-manager': '1.17.6-1.el6'}
2020-05-04 23:18:33,706 [INFO] subscription-manager:1618 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:18:33,707 [INFO] subscription-manager:1618 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=none
2020-05-04 23:18:33,707 [INFO] subscription-manager:1618 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el6', 'subscription-manager': '1.17.6-1.el6'}
2020-05-04 23:18:33,707 [INFO] subscription-manager:1618 @managercli.py:359 - Consumer Identity name=f2808918-3e5c-43e4-bf72-7145a7462903, ip-10-103-136-120 uuid=f2808918-3e5c-43e4-bf72-7145a7462903
2020-05-04 23:18:33,707 [INFO] subscription-manager:1618 @managercli.py:359 - Consumer Identity name=f2808918-3e5c-43e4-bf72-7145a7462903, ip-10-103-136-120 uuid=f2808918-3e5c-43e4-bf72-7145a7462903
2020-05-04 23:18:33,859 [INFO] subscription-manager:1618 @managerlib.py:798 - This consumer's profile has been deleted from the server. Local certificates and cache will be cleaned now.
2020-05-04 23:18:33,862 [INFO] subscription-manager:1618 @managerlib.py:879 - Cleaned local data
2020-05-04 23:18:33,862 [INFO] subscription-manager:1618 @managercli.py:1076 - --force specified, unregistered old consumer: f2808918-3e5c-43e4-bf72-7145a7462903
2020-05-04 23:18:33,863 [INFO] subscription-manager:1618 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=none
2020-05-04 23:18:33,998 [INFO] subscription-manager:1618 @hwprobe.py:908 - collected virt facts: virt.is_guest=True, virt.host_type=xen, xen-hvm, virt.uuid=ec21adad-c480-495a-221e-9cfb805ded40
2020-05-04 23:18:36,852 [INFO] subscription-manager:1618 @managerlib.py:77 - Consumer created: {'consumer_name': '51f15c64-23a9-4d72-ad1e-2e4c020770a1, ip-10-103-136-120', 'uuid': '51f15c64-23a9-4d72-ad1e-2e4c020770a1'}
2020-05-04 23:18:36,859 [INFO] subscription-manager:1618 @connection.py:815 - Connection built: host=forman.here.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2020-05-04 23:18:37,084 [INFO] subscription-manager:1618 @managercli.py:395 - Server Versions: {'rules-version': '5.39', 'candlepin': '3.15.0.1-Unknown', 'server-type': 'Red Hat Subscription Management'}
2020-05-04 23:18:37,793 [INFO] subscription-manager:1618 @managercli.py:1182 - System registered, updating entitlements if needed
2020-05-04 23:18:38,155 [INFO] subscription-manager:1618 @entcertlib.py:131 - certs updated:
Total updates: 1
Found (local) serial# []
Expected (UEP) serial# [484362592190205721]
Added (new)
  [sn:484362592190205721 (Centos6,) @ /etc/pki/entitlement/484362592190205721.pem]
Deleted (rogue):
  <NONE>
2020-05-04 23:18:38,157 [ERROR] subscription-manager:1618 @cache.py:128 - Unable to read cache: /var/lib/rhsm/cache/written_overrides.json
2020-05-04 23:18:38,301 [INFO] subscription-manager:1618 @repolib.py:303 - repos updated: Repo updates

Total repo updates: 5
Updated
    <NONE>
Added (new)
    [id:CHE_Centos6_Centos6_10-FastTrack Centos6.10-FastTrack]
    [id:CHE_Centos6_Centos6_10-OS Centos6.10-OS]
    [id:CHE_Centos6_Centos6_10-Extras Centos6.10-Extras]
    [id:CHE_Centos6_Centos6_10-Updates Centos6.10-Updates]
    [id:CHE_Centos6_Centos6_10-CentosPlus Centos6.10-CentosPlus]
Deleted
    <NONE>

Jonathon_Turel · May 5, 2020, 3:53pm

Hmm. Let’s look at this particular failure:

https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-CentosPlus/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 503"
Trying other mirror.

Try reaching that URL from your browser while running foreman-tail on the Foreman server to confirm that error is coming from Apache.

Then try reaching the same URL but substituting the actual Foreman host name in the URL and see if that works.

Perhaps the ANAME needs to be a SAN on the custom certs you’re using.

reesethegeek · May 5, 2020, 5:39pm

==> /var/log/httpd/foreman-ssl_access_ssl.log <==

5.5.5.5 - 51f15c64-23a9-4d72-ad1e-2e4c020770a1 [05/May/2020:17:22:10 +0000] “GET /rhsm/ HTTP/1.1” 200 2202 “-” “RHSM/1.0 (cmd=yum)”

==> /var/log/foreman/production.log <==
2020-05-05T17:22:10 [I|app|d402da91] Started GET “/rhsm/” for 5.5.5.5 at 2020-05-05 17:22:10 +0000
2020-05-05T17:22:10 [I|app|d402da91] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2020-05-05T17:22:10 [I|app|d402da91] Parameters: {“root”=>{}}
2020-05-05T17:22:10 [I|app|d402da91] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2020-05-05T17:22:10 [I|app|d402da91] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (1.7ms)
2020-05-05T17:22:10 [I|app|d402da91] Completed 200 OK in 15ms (Views: 2.4ms | ActiveRecord: 1.6ms)

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
5.5.5.5 - 51f15c64-23a9-4d72-ad1e-2e4c020770a1 [05/May/2020:17:22:10 +0000] “GET /rhsm/consumers/51f15c64-23a9-4d72-ad1e-2e4c020770a1/content_overrides HTTP/1.1” 200 2 “-” “RHSM/1.0 (cmd=yum)”

==> /var/log/foreman/production.log <==
2020-05-05T17:22:11 [I|app|30dbc1cd] Started GET “/rhsm/consumers/51f15c64-23a9-4d72-ad1e-2e4c020770a1/content_overrides” for 5.5.5.5 at 2020-05-05 17:22:11 +0000
2020-05-05T17:22:11 [I|app|30dbc1cd] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2020-05-05T17:22:11 [I|app|30dbc1cd] Parameters: {“id”=>“51f15c64-23a9-4d72-ad1e-2e4c020770a1”}
2020-05-05T17:22:11 [I|app|30dbc1cd] Completed 200 OK in 40ms (Views: 0.2ms | ActiveRecord: 2.1ms)

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
6.6.6.6 - - [05/May/2020:17:22:13 +0000] “GET /users/login HTTP/1.1” 200 2631 “-” “ELB-HealthChecker/1.0”

==> /var/log/foreman/production.log <==
2020-05-05T17:22:13 [I|app|708485a6] Started GET “/users/login” for 6.6.6.6 at 2020-05-05 17:22:13 +0000
2020-05-05T17:22:13 [I|app|708485a6] Processing by UsersController#login as /
2020-05-05T17:22:13 [I|app|708485a6] Rendering users/login.html.erb within layouts/login
2020-05-05T17:22:13 [I|app|708485a6] Rendered common/_login.html.erb (0.3ms)
2020-05-05T17:22:13 [I|app|708485a6] Rendered users/login.html.erb within layouts/login (0.9ms)
2020-05-05T17:22:13 [I|app|708485a6] Rendering layouts/base.html.erb
2020-05-05T17:22:13 [I|app|708485a6] Rendered layouts/base.html.erb (1.6ms)
2020-05-05T17:22:13 [I|app|708485a6] Completed 200 OK in 9ms (Views: 3.6ms | ActiveRecord: 1.6ms)

reesethegeek · May 5, 2020, 8:22pm

What is also strange is that going to foreman.here.com/users/login works perfectly …but I cannot go to the link above it says in CHOME a Bad CERT error.

The certificate does include the foreman.here.com as a SAN.

reesethegeek · May 6, 2020, 7:23pm

Any ideas?

Jonathon_Turel · May 6, 2020, 9:16pm

Weird. I’m pretty sure you should be seeing those 503s through foreman-tail which your logs do not show. Do you see them anywhere in /var/log/httpd?

In /etc/rhsm/rhsm.conf on your client can you try changing the hostename in ‘baseurl’ under ‘rhsm’ to the actual hostname (not CNAME) of your foreman box?

I’m running out of things to try but the thing to do now is figure out what is responding with a 503

Jonathon_Turel · May 6, 2020, 9:17pm

@katello anyone have any more ideas here?

Justin_Sherrill · May 7, 2020, 3:01pm

Can you try curl’ing the http:// version of that repo?

reesethegeek · May 7, 2020, 3:09pm

@Justin_Sherrill that actually works and returns the expected repomd.xml…thoughts?

<?xml version='1.0' encoding='UTF-8'?>

1588205973
15882059732328659916b823ac57533cacd9a9a4afd16465a8d97c7c3f58387d300bcccedac923dcf39840326b3084f781a9d4f1b132a34d9569fbeb38d63f8fc5d48a88867d0bd6288d0088a
1588205973413144c53866dc7b9159e8256f565618a2f9a00967d32d9721850befb37bb59a50b21f2401224c082aab22760ba082edf816c53443da82610f30b20dfba5e33c034a8c6480c2c
1588205973300182880bedb2a641656ecac0816b626a26b60feb849e95d16d7db5991595eb77cd84f2098154457968fc8fae43e2d1a1de7c0ad443bdc0cc562b3c1ad8913ac250e261c72398d
15882059738643f082c26240935651b16420cfa175f78db26e78a8b5e3af08fd0b2fafc285b2477102284a0c31c604db32ab2af23d5c83938ccfce2de45acfe40ec912e27ef48556236
158820597392d6bf5e5ac1a942454ecf953c8c30c93d6110a4853f0492e9a6826ea5f17b6c1251f5922ee6f9f76089a8135e281dbd0e88ae4765166a8b9c66659de709cc556f08
1588205973124a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70

Justin_Sherrill · May 7, 2020, 3:26pm

It definitely seems like an SSL related issue then. Can you provide the output of:

curl -vvvv https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-CentosPlus/repodata/repomd.xml

reesethegeek · May 7, 2020, 4:36pm

curl -vvvv https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-CentosPlus/repodata/repomd.xml

* About to connect() to foreman.uscis.dhs.gov port 443 (#0)

* Trying 5.5.5.5... connected

* Connected to foreman.here.com (5.5.5.5) port 443 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

* CAfile: /etc/pki/tls/certs/ca-bundle.crt

CApath: none

* NSS: client certificate not found (nickname not specified)

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

* subject: CN=foreman.here.com,OU=Devices,OU=COMPANY,OU=REMOVED DUE TO SENSITIVITY,O=REMOVED DUE TO SENSITIVITY,C=US

* start date: Apr 27 22:36:00 2020 GMT

* expire date: Jul 27 23:06:00 2022 GMT

* common name: foreman.here.com

* issuer: OU=AGENCY CA4,OU=Certification Authorities,OU=REMOVED DUE TO SENSENTIVITY,O=SENSITIVITY,C=US

> GET /pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-CentosPlus/repodata/repomd.xml HTTP/1.1

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: foreman.here.com

> Accept: */*

> 

* NSS: client certificate not found (nickname not specified)

* SSL read: errno -12227

* Closing connection #0

curl: (56) NSS: client certificate not found (nickname not specified)

Justin_Sherrill · May 7, 2020, 5:09pm

Thats actually probably expected, but it may not be getting far enough in the stack without some client certs, so try this on a client that is subscribed:

first look in /etc/pki/entitlement/ for a cert/key pair, for example:
2039364093364475299-key.pem 2039364093364475299.pem

curl -vvvv https://foreman.here.com/pulp/repos/CHE/Library/Centos6_Content_View/custom/Centos6/Centos6_10-CentosPlus/repodata/repomd.xml   --cert /etc/pki/entitlement/XXXX.pem  --key /etc/pki/entitlement/XXXX-key.pem

and let me know what that shows.

reesethegeek · May 7, 2020, 5:40pm

Figured out the issue…and now I feel bad
The proxy was set in the /etc/yum.conf and was sending the yum traffic out side instead of inside.

I removed
proxy=http://45.45.45.45:80
from /etc/yum.conf

cleared the registraiton from the machine using the subscription-mananger command
removed the yum cache from the system
removed the katello.rpm from the system
installed again the katello package from the foreman server
re-registered the host to the forman server using the subscription-manager command
and then was successfully able to pull the packages.

I want to give a special shoutout to both @Justin_Sherrill and @Jonathon_Turel for helping me out on this. Much Much Appreciated!!!

Justin_Sherrill · May 7, 2020, 5:57pm

Awesome! Glad you figured it out!

Jonathon_Turel · May 7, 2020, 6:06pm

Thanks for following up with the root cause. That really helps us too!