Another ssl error since updating my certs

Support
1

Problem:
After updating the custom certificates for my foreman instance and hitting a few bugs along the way, my system still has problems, but hopefully i’ve found the last one: Puma

httpd foreman-ssl_error_ssl.log:

[Tue Apr 06 15:17:48.838386 2021] [proxy:error] [pid 29465] [client 10.100.128.153:38966] AH00898: Error reading from remote server returned by /rhsm/consumers/b5ca13dc-5dc5-4120-96b2-8e823afc16dc
[Tue Apr 06 15:22:08.810064 2021] [ssl:error] [pid 29465] [client 10.110.134.14:33962] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain
[Tue Apr 06 15:22:09.948190 2021] [ssl:error] [pid 31441] [client 10.110.134.14:33964] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain
[Tue Apr 06 15:24:28.296187 2021] [proxy_http:error] [pid 31805] (70007)The timeout specified has expired: [client 10.101.228.206:54478] AH01102: error reading status line from remote server 127.0.0.1:3000
[Tue Apr 06 15:24:28.296224 2021] [proxy:error] [pid 31805] [client 10.101.228.206:54478] AH00898: Error reading from remote server returned by /rhsm/consumers/2d1dd324-b74d-46c8-bc03-e292b7c77fc1

Now, my main problem with finding these errors, is that i have absolutely no idea what backend service is getting the error or where the, probably broken, certificates are stored.

So, full story:
Foreman certificate was expiring, updated it according to documentation, reran installer to handle the actual certification process and everything appeared to be OK. Couple days later we start noticing weird behaviour with updates and Foreman, and it turned out we had hit the “fun” bug of candlepin’s truststore not being updated properly. Fixed that and the system got a bit better, managed to remain usable for a couple hours at a time, found some more ssl errors and ended up testing out deleting and replacing a whole bunch of certificates until the errors stopped (one certificate at a time, reverting to snapshots when messing about broke things more).
The system was installed a couple years ago using katello 3.13 and has, for the most part, worked flawlessly until the cert expired and was swapped out in beginning of February, since then it’s been one ssl error after the next, but hopefully this is the last one and i can move onto my pulp2 -> pulp3 migration errors before the 4.0 release.

Expected outcome:
No ssl errors in httpd logs

Foreman and Proxy versions:
Katello 3.18.2
All latest updates have been applied

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

2

Perhaps should be noted that when Foreman hangs and becomes unresponsive, a simple ‘systemctl restart tomcat’ fixes the issue.

3

I think i’ve found at least A problem if not THE problem. /etc/candlepin/certs/amqp/keystore did not get updated when the cert was renewed, so it no longer matches the client certs.
How would i go about recreating or adding the correct cert into the keystore?
I’ve tried removing it and rerunning installer, but that did nothing, so i probably need some magic command or do it manually.

4

An update was just release for qdrouterd - that appears to have removed the ssl errors
Also turns out that my root partition was at 92% which caused qdrouter to stop writing to disk, even though there was plenty of space left ( found that default value appears to be max-disk: 90% )
After fixing that, i can’t really find any errors, but will see throughout the day if anything crashes again.

I’ve decided to be Mr. Hopeful that this is now fixed.

1 Like