Problem:
System registers against proxy but fails to upload packages. Receives a 401 from Server. It may have something to do with SSL certs, as I’ve created a bit of a monster, but can’t see anything specific in the logs anymore. The proxy also sits behind a reverse proxy, which filters allowed requests, mapping from an external domain to an internal domain.
Expected outcome:
System registers against proxy and uploads package information.
Foreman and Proxy versions:
3.4 both Server and Proxy.
Foreman and Proxy plugin versions:
Distribution and version:
Other relevant data:
Our Proxy requests:
1663362709.784 6123 X.X.X.X TCP_MISS/200 41930 POST https://foreman/rhsm/consumers?owner=XXX&activation_keys=XXX - FIRSTUP_PARENT/XXX application/json
1663362710.155 285 X.X.X.X TCP_MISS/200 1640 GET https://XXX/rhsm/status - FIRSTUP_PARENT/XXX application/json
1663362710.293 65 X.X.X.X TCP_MISS/401 1029 GET https://foreman/rhsm/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b - FIRSTUP_PARENT/X.X.X.X application/json
Host:
CentOS 7:
succeeds registration, presents error:
Unauthorized: Invalid credentials for request.
CentOS 6:
succeeds, registration, but doesn’t present an error for package sync
production log:
2022-09-16T22:11:50 [I|app|d7a8b9a9] Started GET “/rhsm/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b” for X.X.X.X at 2022-09-16 22:11:50 +0100
2022-09-16T22:11:50 [I|app|d7a8b9a9] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_show as JSON
2022-09-16T22:11:50 [I|app|d7a8b9a9] Parameters: {“id”=>“a9d72242-1802-4b06-a37e-ede8cd24887b”}
2022-09-16T22:11:50 [I|app|d7a8b9a9] Rendered api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout (Duration: 2.4ms | Allocations: 2581)
2022-09-16T22:11:50 [I|app|d7a8b9a9] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 3.8ms | Allocations: 5137)
2022-09-16T22:11:50 [I|app|d7a8b9a9] Filter chain halted as :authorize_client_or_user rendered or redirected
2022-09-16T22:11:50 [I|app|d7a8b9a9] Completed 401 Unauthorized in 20ms (Views: 4.8ms | ActiveRecord: 6.9ms | Allocations: 9224)
Candlepin:
grep a9d72242-1802-4b06-a37e-ede8cd24887b /var/log/candlepin/candlepin.log -A1 | sed G
2022-09-16 22:11:48,099 [thread=https-jsse-nio-127.0.0.1-23443-exec-8] [req=a14393ef-d4c9-4d52-a550-f64260abb741, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.resource.ConsumerResource - Consumer a9d72242-1802-4b06-a37e-ede8cd24887b created in org 2c9c69e48200fc370182188cd7bb01cf
2022-09-16 22:11:48,142 [thread=https-jsse-nio-127.0.0.1-23443-exec-8] [req=a14393ef-d4c9-4d52-a550-f64260abb741, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=3253
2022-09-16 22:11:49,243 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=59698a58-e9ae-4d2b-b261-bf8ae6af2de2, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b
2022-09-16 22:11:49,419 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=59698a58-e9ae-4d2b-b261-bf8ae6af2de2, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=176
2022-09-16 22:11:49,467 [thread=https-jsse-nio-127.0.0.1-23443-exec-9] [req=33405bf6-99cc-4d9a-9fc6-fc7c828caeb8, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b/guests
2022-09-16 22:11:49,490 [thread=https-jsse-nio-127.0.0.1-23443-exec-9] [req=33405bf6-99cc-4d9a-9fc6-fc7c828caeb8, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=23
2022-09-16 22:11:49,516 [thread=https-jsse-nio-127.0.0.1-23443-exec-5] [req=588efcca-d23b-4d2b-87e1-77c825e3ede0, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b/host
2022-09-16 22:11:49,555 [thread=https-jsse-nio-127.0.0.1-23443-exec-5] [req=588efcca-d23b-4d2b-87e1-77c825e3ede0, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=400, content-type=“application/json”, time=39
2022-09-16 22:11:49,661 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=5507f2e1-56f7-48a2-a761-ea31c5493cc6, org=, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/a9d72242-1802-4b06-a37e-ede8cd24887b
2022-09-16 22:11:49,739 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=5507f2e1-56f7-48a2-a761-ea31c5493cc6, org=XXX, csid=b4d99d91-789e-4a88-bb8e-fad05ce814d4] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type=“application/json”, time=78
grep a9d72242-1802-4b06-a37e-ede8cd24887b /var/log/candlepin/audit.log
2022-09-16 22:11:48,143 principalType=trusteduser principal=foreman_admin target=COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“reasons”:,“status”:“valid”}
2022-09-16 22:11:48,145 principalType=trusteduser principal=foreman_admin target=SYSTEM_PURPOSE_COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“nonCompliantUsage”:null,“compliantAddOns”:{},“nonCompliantRole”:null,“reasons”:,“nonCompliantServiceType”:null,“compliantSLA”:{},“nonCompliantAddOns”:,“compliantRole”:{},“nonCompliantSLA”:null,“compliantUsage”:{},“status”:“not specified”,“compliantServiceType”:{}}
2022-09-16 22:11:48,148 principalType=trusteduser principal=foreman_admin target=COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“reasons”:,“status”:“valid”}
2022-09-16 22:11:48,151 principalType=trusteduser principal=foreman_admin target=SYSTEM_PURPOSE_COMPLIANCE entityId=a9d72242-1802-4b06-a37e-ede8cd24887b type=CREATED owner=2c9c69e48200fc370182188cd7bb01cf eventData={“nonCompliantUsage”:null,“compliantAddOns”:{},“nonCompliantRole”:null,“reasons”:,“nonCompliantServiceType”:null,“compliantSLA”:{},“nonCompliantAddOns”:,“compliantRole”:{},“nonCompliantSLA”:null,“compliantUsage”:{},“status”:“not specified”,“compliantServiceType”:{}}