Candlepin not trusting CA after cert update

Problem:
2021-03-12T13:59:59 [E|app|ca06f9ee] Error occurred while starting Katello::CandlepinEventListener
2021-03-12T13:59:59 [E|app|ca06f9ee] SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown

Expected outcome:
No error.

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

Foreman/Katello has been running (mostly) smooth for 2 years now, but the server certificate was expiring so we updated it and ran the appropriate commands. Everything appeared to work fine, until i updated to latest Patch level and Foreman got properly restarted.
Should be noted here that the new certificate has the same Root, but a new Intermediate root (issuer).
First problem I encountered was subscription manager got a CA error on all 260 hosts, I ended up having katello recreate the pulp certs and ran a full clean and re-register on all 260 hosts - Everything appeared to work fine after that.
However, a couple days later we started noticing that Foreman would grind to a halt after a few hours and after tracing logs, the issue appears to be this Candlepin error causing hangups ( it’s at least the only error in the logs )
I probably messed something up along the way through this process, but for some reason everything works Except the system doesn’t appear to trust the Candlepin cert and i can’t figure out how to force katello to recreate it or fix this issue.

Any help would be greatly appreciated.
Absolute worst case scenario, i’d have to restore from backups since mid February and try again, discarding every parameter change and new host that has been created since

~]# hammer ping
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 18ms
candlepin_events:
Status: FAIL
message: Not running
Server Response: Duration: 0ms
candlepin_auth:
Status: ok
Server Response: Duration: 16ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
pulp:
Status: ok
Server Response: Duration: 64ms
pulp_auth:
Status: ok
Server Response: Duration: 22ms
foreman_tasks:
Status: ok
Server Response: Duration: 3ms

It is possible you are running into: Bug #31574: The Artemis client certificate is not updated in truststore if it changes - Installer - Foreman

You can try removing /etc/candlepin/certs/truststore and re-running the installer. That should in theory re-build the truststore properly for your current certs.

2 Likes

Looks very likely, and just as i read your message i am attempting exactly that :slight_smile:

And it’s working again. thanks :slight_smile:

3 Likes