Katello::CandlepinEventListener not starting after upgrade to Foreman 2.2.2/Katello 3.17

Support
1

Problem:

After upgrade from Foreman 2.0 / Katello 3.15, the Katello::CandlepinEventListener is reporting the following errors in /var/log/foreman/production.log.

2021-01-05T08:58:46 [E|app|2ff80e80] Error occurred while starting Katello::CandlepinEventListener
2021-01-05T08:58:46 [E|app|2ff80e80] SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown
2021-01-05T08:58:46 [E|app|2ff80e80] /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/connection/netio.rb:465:in `connect'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/connection/netio.rb:465:in `block in open_ssl_socket'
 2ff80e80 | /opt/rh/rh-ruby25/root/usr/share/ruby/timeout.rb:76:in `timeout'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/connection/netio.rb:460:in `open_ssl_socket'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/connection/netio.rb:520:in `open_socket'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/connection/utils.rb:116:in `block in socket'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/connection/utils.rb:109:in `synchronize'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/connection/utils.rb:109:in `socket'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/stomp/connection.rb:173:in `initialize'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/stomp/client.rb:134:in `new'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/stomp/client.rb:134:in `create_connection'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/stomp/client.rb:101:in `block in initialize'
 2ff80e80 | /opt/rh/rh-ruby25/root/usr/share/ruby/timeout.rb:93:in `block in timeout'
 2ff80e80 | /opt/rh/rh-ruby25/root/usr/share/ruby/timeout.rb:33:in `block in catch'
 2ff80e80 | /opt/rh/rh-ruby25/root/usr/share/ruby/timeout.rb:33:in `catch'
 2ff80e80 | /opt/rh/rh-ruby25/root/usr/share/ruby/timeout.rb:33:in `catch'
 2ff80e80 | /opt/rh/rh-ruby25/root/usr/share/ruby/timeout.rb:108:in `timeout'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/stomp/client.rb:99:in `initialize'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/lib/katello/messaging/stomp_connection.rb:69:in `new'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/lib/katello/messaging/stomp_connection.rb:69:in `client'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/lib/katello/messaging/stomp_connection.rb:43:in `subscribe'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/candlepin_event_listener.rb:37:in `run'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:33:in `block in check_services'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:23:in `each'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:23:in `check_services'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:16:in `block (2 levels) in start'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:15:in `block in start'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:14:in `loop'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:14:in `start'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/app/services/katello/event_daemon.rb:119:in `block in start_monitor_thread'
 2ff80e80 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Per other posts in the support forum, I have verified that the tomcat service is up and running and I am able to communicate with it via localhost on the server (openssl s_client -connect localhost:8443 | openssl x509 -text).

All other functions on the servers appear to be working normally. Per the output from the above troubleshooting step, I am using the self-sign certificate on the tomcat service. What further steps can I do to continue to troubleshoot this problem and work towards a solution?

2

We think this is related to an issue with the truststore not getting updated when a certificate gets updated. Could you try the following:

  1. rm /etc/candlepin/certs/truststore
  2. foreman-installer

This should force the re-build of the truststore with the latest certificates. We are aiming to fix this officially: https://github.com/theforeman/puppet-certs/pull/311

3

Thanks for the response, but I decided to do a rebuild of the server to cleanup things that might have crept into the server over the last year. I will make note of your suggestion if I do encounter this issue going forward and give it a go.

4

FYI this procedure worked for me. Iā€™d recently updated my certificate and ran across this exact issue.

2 Likes