SmartProxy host registration 3.4

Problem:
System registers against proxy but fails to upload packages. Receives a 401 from Server. It may have something to do with SSL certs, as I’ve created a bit of a monster, but can’t see anything specific in the logs anymore. The proxy also sits behind a reverse proxy, which filters allowed requests, mapping from an external domain to an internal domain.

Expected outcome:
System registers against proxy and uploads package information.

Foreman and Proxy versions:
3.4 both Server and Proxy.

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:
Our Proxy requests:
1663362709.784 6123 X.X.X.X TCP_MISS/200 41930 POST https://foreman/rhsm/consumers?owner=XXX&activation_keys=XXX - FIRSTUP_PARENT/XXX application/json
1663362710.155 285 X.X.X.X TCP_MISS/200 1640 GET https://XXX/rhsm/status - FIRSTUP_PARENT/XXX application/json
1663362710.293 65 X.X.X.X TCP_MISS/401 1029 GET https://foreman/rhsm/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b - FIRSTUP_PARENT/X.X.X.X application/json

Host:
CentOS 7:
succeeds registration, presents error:
Unauthorized: Invalid credentials for request.
CentOS 6:
succeeds, registration, but doesn’t present an error for package sync

production log:
2022-09-16T22:11:50 [I|app|d7a8b9a9] Started GET “/rhsm/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b” for X.X.X.X at 2022-09-16 22:11:50 +0100
2022-09-16T22:11:50 [I|app|d7a8b9a9] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_show as JSON
2022-09-16T22:11:50 [I|app|d7a8b9a9] Parameters: {“id”=>“a9d72242-1802-4b06-a37e-ede8cd24887b”}
2022-09-16T22:11:50 [I|app|d7a8b9a9] Rendered api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout (Duration: 2.4ms | Allocations: 2581)
2022-09-16T22:11:50 [I|app|d7a8b9a9] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 3.8ms | Allocations: 5137)
2022-09-16T22:11:50 [I|app|d7a8b9a9] Filter chain halted as :authorize_client_or_user rendered or redirected
2022-09-16T22:11:50 [I|app|d7a8b9a9] Completed 401 Unauthorized in 20ms (Views: 4.8ms | ActiveRecord: 6.9ms | Allocations: 9224)

Candlepin:
grep a9d72242-1802-4b06-a37e-ede8cd24887b /var/log/candlepin/candlepin.log -A1 | sed G
2022-09-16 22:11:48,099 [thread=https-jsse-nio-127.0.0.1-23443-exec-8] [req=a14393ef-d4c9-4d52-a550-f64260abb741, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.resource.ConsumerResource - Consumer a9d72242-1802-4b06-a37e-ede8cd24887b created in org 2c9c69e48200fc370182188cd7bb01cf

2022-09-16 22:11:48,142 [thread=https-jsse-nio-127.0.0.1-23443-exec-8] [req=a14393ef-d4c9-4d52-a550-f64260abb741, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=3253

2022-09-16 22:11:49,243 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=59698a58-e9ae-4d2b-b261-bf8ae6af2de2, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b

2022-09-16 22:11:49,419 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=59698a58-e9ae-4d2b-b261-bf8ae6af2de2, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=176

2022-09-16 22:11:49,467 [thread=https-jsse-nio-127.0.0.1-23443-exec-9] [req=33405bf6-99cc-4d9a-9fc6-fc7c828caeb8, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b/guests

2022-09-16 22:11:49,490 [thread=https-jsse-nio-127.0.0.1-23443-exec-9] [req=33405bf6-99cc-4d9a-9fc6-fc7c828caeb8, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=23

2022-09-16 22:11:49,516 [thread=https-jsse-nio-127.0.0.1-23443-exec-5] [req=588efcca-d23b-4d2b-87e1-77c825e3ede0, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b/host

2022-09-16 22:11:49,555 [thread=https-jsse-nio-127.0.0.1-23443-exec-5] [req=588efcca-d23b-4d2b-87e1-77c825e3ede0, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=400, content-type=“application/json”, time=39

2022-09-16 22:11:49,661 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=5507f2e1-56f7-48a2-a761-ea31c5493cc6, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b

2022-09-16 22:11:49,739 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=5507f2e1-56f7-48a2-a761-ea31c5493cc6, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=78

grep a9d72242-1802-4b06-a37e-ede8cd24887b /var/log/candlepin/audit.log

2022-09-16 22:11:48,143 principalType=trusteduser principal=foreman_admin target=COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“reasons”:,“status”:“valid”}
2022-09-16 22:11:48,145 principalType=trusteduser principal=foreman_admin target=SYSTEM_PURPOSE_COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“nonCompliantUsage”:null,“compliantAddOns”:{},“nonCompliantRole”:null,“reasons”:,“nonCompliantServiceType”:null,“compliantSLA”:{},“nonCompliantAddOns”:,“compliantRole”:{},“nonCompliantSLA”:null,“compliantUsage”:{},“status”:“not specified”,“compliantServiceType”:{}}
2022-09-16 22:11:48,148 principalType=trusteduser principal=foreman_admin target=COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“reasons”:,“status”:“valid”}
2022-09-16 22:11:48,151 principalType=trusteduser principal=foreman_admin target=SYSTEM_PURPOSE_COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“nonCompliantUsage”:null,“compliantAddOns”:{},“nonCompliantRole”:null,“reasons”:,“nonCompliantServiceType”:null,“compliantSLA”:{},“nonCompliantAddOns”:,“compliantRole”:{},“nonCompliantSLA”:null,“compliantUsage”:{},“status”:“not specified”,“compliantServiceType”:{}}

maybe not SSL certs? possibly something else to do with my proxy chain?
I’ve definitely run out of ideas for today, so some direction will be much appreciated! :slight_smile:

Right, I’ve successfully got packages synced with an internal host. I know how much Foreman cares about domains being right, so probably something in my crazy setup needs looking at.

For anyone wondering (probably not), it’ll probably be these, as I’m getting 401 authenticating after registration:
/etc/httpd/conf.d/10-rhsm-pulpcore-https-443.conf: SSLProxyMachineCertificateFile “/etc/pki/katello/private/something-foreman-proxy-client-bundle.pem”

RequestHeader set X_RHSM_SSL_CLIENT_CERT “%{SSL_CLIENT_CERT}s”

I’ll look at my weird setup when I have time.