Problem:
When attempting to register hosts, whether via subscription-manager or using the curl script generated within foreman, I receive an error “Unauthorized: Invalid credentials for request”. When using subscription manager I have tried to register a test host with an activation key and using interactive registration with my admin credentials. The host does appear to be added to the foreman instance as it shows up in content hosts with proper environment and view, but it appears katello/candlepin are preventing it from seeing its available repos.
Expected outcome:
Host will be registered and able to see associated repositories
Foreman and Proxy versions:
foreman v3.3.1, katello v4.5.1
Foreman and Proxy plugin versions:
Distribution and version:
Foreman on CentOS 7.9
Test host on CentOS 7.9
Other relevant data:
production.log relevant output on foreman instance:
2023-03-07T21:04:10 [I|app|b7e494eb] Started GET “/rhsm/status” for 10.x.x.x at 2023-03-07 21:04:10 +0000
2023-03-07T21:04:10 [I|app|b7e494eb] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2023-03-07T21:04:10 [I|app|b7e494eb] Completed 200 OK in 33ms (Views: 0.3ms | ActiveRecord: 5.8ms | Allocations: 5804)
2023-03-07T21:04:10 [I|app|d64409a8] katello event handled success=true type=import_pool object_id=2 expired=false rescheduled=false duration=100.72
2023-03-07T21:04:11 [I|app|939b0388] Started GET “/rhsm/consumers/2a8d47db-f370-488d-ba2e-6719a2a17618” for 10.x.x.x at 2023-03-07 21:04:11 +0000
2023-03-07T21:04:11 [I|app|939b0388] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_show as JSON
2023-03-07T21:04:11 [I|app|939b0388] Parameters: {“id”=>“2a8d47db-f370-488d-ba2e-6719a2a17618”}
2023-03-07T21:04:11 [I|app|939b0388] Rendering api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout
2023-03-07T21:04:11 [I|app|939b0388] Rendered api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout (Duration: 0.8ms | Allocations: 323)
2023-03-07T21:04:11 [I|app|939b0388] Filter chain halted as :authorize_client_or_user rendered or redirected
2023-03-07T21:04:11 [I|app|939b0388] Completed 401 Unauthorized in 9ms (Views: 2.4ms | ActiveRecord: 2.5ms | Allocations: 2936)
subscription-manager register --org=“My Org” --force (From test host):
Username: user with admin permissions
Password:
Hint: Organization “My Org” contains following environments: accurate list of content views
Unauthorized: Invalid credentials for request.
Similar error can be seen with “subscription-manager list --available (–all)”
/var/log/rhsm/rhsm.log on test host
2023-03-07 21:14:28,018 [INFO] subscription-manager:71164:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:28,027 [INFO] subscription-manager:71164:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:28,027 [INFO] subscription-manager:71164:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=none
2023-03-07 21:14:34,949 [INFO] subscription-manager:71164:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=basic username=username with admin
2023-03-07 21:14:41,254 [INFO] yum.py:71227:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:41,254 [INFO] yum.py:71227:MainThread entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial#
Expected (UEP) serial#
Added (new)
Deleted (rogue):
2023-03-07 21:14:46,736 [INFO] yum.py:71823:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:46,737 [INFO] yum.py:71823:MainThread entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial#
Expected (UEP) serial#
Added (new)
Deleted (rogue):
2023-03-07 21:14:48,498 [INFO] yum:71834:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:48,498 [INFO] yum:71834:MainThread entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial#
Expected (UEP) serial#
Added (new)
Deleted (rogue):
2023-03-07 21:14:51,748 [INFO] repoquery:71889:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:51,748 [INFO] repoquery:71889:MainThread entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial#
Expected (UEP) serial#
Added (new)
Deleted (rogue):
2023-03-07 21:14:52,621 [INFO] repoquery:71900:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:52,622 [INFO] repoquery:71900:MainThread entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial#
Expected (UEP) serial#
Added (new)
Deleted (rogue):
2023-03-07 21:14:58,219 [INFO] subscription-manager:71164:MainThread managerlib.py:72 - Consumer created: testhost.domain.com (2a8d47db-f370-488d-ba2e-6719a2a17618)
2023-03-07 21:14:58,220 [INFO] subscription-manager:71164:MainThread connection.py:915 - Connection built: host=myforeman-instance port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2023-03-07 21:14:58,958 [ERROR] subscription-manager:71164:MainThread connection.py:647 - Response: 401
2023-03-07 21:14:58,958 [ERROR] subscription-manager:71164:MainThread connection.py:648 - JSON parsing error: Expecting ‘:’ delimiter: line 1 column 9 (char 8)
2023-03-07 21:14:58,958 [ERROR] subscription-manager:71164:MainThread managercli.py:217 - Error during registration: Server error attempting a GET to /rhsm/consumers/2a8d47db-f370-488d-ba2e-6719a2a17618 returned status 401
Unauthorized: Invalid credentials for request.
2023-03-07 21:14:58,958 [ERROR] subscription-manager:71164:MainThread managercli.py:218 - Server error attempting a GET to /rhsm/consumers/2a8d47db-f370-488d-ba2e-6719a2a17618 returned status 401
Unauthorized: Invalid credentials for request.
Traceback (most recent call last):
File “/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py”, line 1389, in _do_command
type=self.options.consumertype
File “/usr/lib64/python2.7/site-packages/rhsmlib/services/register.py”, line 106, in register
store.sync()
File “/usr/lib/python2.7/site-packages/syspurpose/files.py”, line 281, in sync
remote_contents = self.get_remote_contents()
File “/usr/lib/python2.7/site-packages/syspurpose/files.py”, line 344, in get_remote_contents
consumer = self.uep.getConsumer(self.consumer_uuid)
File “/usr/lib64/python2.7/site-packages/rhsm/connection.py”, line 1237, in getConsumer
return self.conn.request_get(method)
File “/usr/lib64/python2.7/site-packages/rhsm/connection.py”, line 730, in request_get
return self._request(“GET”, method, headers=headers)
File “/usr/lib64/python2.7/site-packages/rhsm/connection.py”, line 756, in _request
info=info, headers=headers)
File “/usr/lib64/python2.7/site-packages/rhsm/connection.py”, line 631, in _request
self.validateResponse(result, request_type, handler)
File “/usr/lib64/python2.7/site-packages/rhsm/connection.py”, line 703, in validateResponse
handler=handler)
UnauthorizedException: Server error attempting a GET to /rhsm/consumers/2a8d47db-f370-488d-ba2e-6719a2a17618 returned status 401
Unauthorized: Invalid credentials for request.
Content of /etc/pki/consumer/cert.pem on test host:
rct cat-cert /etc/pki/consumer/cert.pem
±------------------------------------------+
Identity Certificate
±------------------------------------------+
Certificate:
Path: /etc/pki/consumer/cert.pem
Version: 1.0
Serial: 6664726946112879068
Start Date: 2023-03-07 20:14:57+00:00
End Date: 2039-03-07 21:14:57+00:00
Alt Name: DirName:/O=MyOrg/CN=2a8d47db-f370-488d-ba2e-6719a2a17618, DirName:/CN=testhost.domain.com
Subject:
CN: 2a8d47db-f370-488d-ba2e-6719a2a17618
O: MyOrg
Issuer:
C: US
CN: myforeman-instance
L: MyCity
O: Katello
OU: SomeOrgUnit
ST: MyState
This is all after having installed katello-ca-consumer-latest.noarch.rpm, confirmed host name is correct in /etc/rhsm/rhsm.conf.
This Foreman instance is fronted by an AWS ALB, the hostname was set to match the CN for the ALB before installing foreman/katello. The ALB uses a custom SSL certificate from our CA chain that matches the hostname configured on the instance but all certs on the host are the default as generated during install. I confirmed that X-Forwarded-For is properly being passed by the ALB, however I noticed that the IP I have obfuscated in log output is the IP of the ALB and not the host so that may be relevant.