Candlepin_events FAIL, user authentication error

Problem:
On my Foreman server, candlepin_events fails to start. hammer ping shows:

database:         
    Status:          ok
    Server Response: Duration: 1ms
cache:            
    servers: 
     1) Status:          ok
        Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 26ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 15ms
candlepin_events: 
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:            
    Status:          ok
    Server Response: Duration: 387ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 64ms
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 6ms

And /var/log/candlepin/error.log shows errors related to user authentication:

2024-09-10 16:18:04,575 [thread=Thread-5 (activemq-netty-threads)] [=, org=, csid=] 
WARN  org.apache.activemq.artemis.core.server - 
AMQ222216: Security problem while authenticating: 
AMQ229031: Unable to validate user from 127.0.0.1:37544. 
Username: null; SSL certificate subject DN: CN=<*myhost.mydomain*>, OU=PUPPET, O=FOREMAN, ST=North Carolina, C=US

2024-09-10 16:18:04,576 [thread=Thread-5 (activemq-netty-threads)] [=, org=, csid=] 
WARN  org.apache.activemq.artemis.core.protocol.stomp - 
AMQ332069: Sent ERROR frame to STOMP client 127.0.0.1:37544: 
Security Error occurred: User name [null] or password is invalid

I’m not sure if it’s related, but when I look at the keystore file for candlepin, it only shows a single entry, the private key:

keytool -list -keystore /etc/candlepin/certs/keystore
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

tomcat, Aug 30, 2024, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): BF:EF:3B:41:7C:31:A5:C4:D1:E0:7D:F4:A6:02:24:BD:BA:71:F3:95:21:13:75:16:2F:86:F2:B2:05:A0:6F:C5

What have I missed?

Foreman and Proxy versions:

Foreman 3.11
Katello 4.13
Candlepin 4.4

Distribution and version:

Alma Linux 8.10

Is the tomcat service running?

Any errors about candlepin events in /var/log/foreman/production.log?

Yes, tomcat is running:

# systemctl status tomcat
â—Ź tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-09-12 15:08:27 EDT; 10min ago
 Main PID: 377387 (java)

There are errors in /var/log/foreman/production.log, but I can’t interpret them:

2024-09-12T15:16:28 [E|app|8af21059] Error occurred while starting Katello::CandlepinEventListener
2024-09-12T15:16:28 [E|app|8af21059] no current connection exists
2024-09-12T15:16:28 [E|app|8af21059] /usr/share/gems/gems/stomp-1.4.10/lib/stomp/connection.rb:349:in `subscribe'
 8af21059 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:190:in `subscribe'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:43:in `subscribe'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/services/katello/candlepin_event_listener.rb:34:in `run'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:40:in `block in check_services'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `each'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `check_services'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:16:in `block (2 levels) in start'
 8af21059 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:87:in `wrap'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:15:in `block in start'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `loop'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `start'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:84:in `block (2 levels) in start_monitor_thread'
 8af21059 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
 8af21059 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:83:in `block in start_monitor_thread'
 8af21059 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
.
.
.
2024-09-12T15:18:39 [E|app|2766e2ef] Error occurred while starting Katello::CandlepinEventListener
2024-09-12T15:18:39 [E|app|2766e2ef] no current connection exists
2024-09-12T15:18:39 [E|app|2766e2ef] /usr/share/gems/gems/stomp-1.4.10/lib/stomp/connection.rb:349:in `subscribe'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:190:in `subscribe'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:43:in `subscribe'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/services/katello/candlepin_event_listener.rb:34:in `run'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:40:in `block in check_services'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `each'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `check_services'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:16:in `block (2 levels) in start'
 2766e2ef | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:87:in `wrap'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:15:in `block in start'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `loop'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `start'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:84:in `block (2 levels) in start_monitor_thread'
 2766e2ef | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:83:in `block in start_monitor_thread'
 2766e2ef | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-09-12T15:18:54 [E|app|2766e2ef] Error occurred while starting Katello::CandlepinEventListener
2024-09-12T15:18:54 [E|app|2766e2ef] SSL_write
2024-09-12T15:18:54 [E|app|2766e2ef] /usr/share/ruby/openssl/buffering.rb:322:in `syswrite'
 2766e2ef | /usr/share/ruby/openssl/buffering.rb:322:in `do_write'
 2766e2ef | /usr/share/ruby/openssl/buffering.rb:409:in `puts'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:309:in `_wire_write'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:262:in `block (2 levels) in _transmit'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:256:in `each'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:256:in `block in _transmit'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:236:in `synchronize'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:236:in `_transmit'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:208:in `transmit'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/connection.rb:371:in `subscribe'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:190:in `subscribe'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:43:in `subscribe'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/services/katello/candlepin_event_listener.rb:34:in `run'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:40:in `block in check_services'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `each'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `check_services'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:16:in `block (2 levels) in start'
 2766e2ef | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:87:in `wrap'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:15:in `block in start'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `loop'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `start'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:84:in `block (2 levels) in start_monitor_thread'
 2766e2ef | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:83:in `block in start_monitor_thread'
 2766e2ef | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-09-12T15:19:09 [E|app|2766e2ef] Error occurred while starting Katello::CandlepinEventListener
2024-09-12T15:19:09 [E|app|2766e2ef] no current connection exists
2024-09-12T15:19:09 [E|app|2766e2ef] /usr/share/gems/gems/stomp-1.4.10/lib/stomp/connection.rb:349:in `subscribe'
 2766e2ef | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:190:in `subscribe'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:43:in `subscribe'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/services/katello/candlepin_event_listener.rb:34:in `run'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:40:in `block in check_services'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `each'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `check_services'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:16:in `block (2 levels) in start'
 2766e2ef | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:87:in `wrap'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:15:in `block in start'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `loop'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `start'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:84:in `block (2 levels) in start_monitor_thread'
 2766e2ef | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
 2766e2ef | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:83:in `block in start_monitor_thread'
 2766e2ef | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
.
.
.

Looks like it may be some sort of SSL error? Are you using custom certificates?

perhaps running foreman-installer will help but I’m not sure how to troubleshoot this further…

Yes, we’re using custom certs from Let’s Encrypt for this, and the whole SSL setup process for Foreman/Katello/Puppet/Candlepin has my head spinning. I’m going to try a clean setup with foreman-installer without wiping the database we already have.

This is working now. The issue was in the certificate chain for our Let’s Encrypt certs. There is a CA cert installed with it, but it was not enough for openssl verify to verify it. I needed to add CA certs to it from the system-wide CA file. After that, hammer ping shows the status of candlepin_events as OK.

1 Like

Well, it was working, and now it’s not again. Once again, hammer ping is showing

candlepin_events: 
    Status:          FAIL
    message:         Not running

/var/log/candlepin/error.log shows errors about being unable to validate the user:

2024-09-23 11:01:01,301 [thread=Thread-9 (activemq-netty-threads)] [=, org=, csid=] 
WARN  org.apache.activemq.artemis.core.server - AMQ222216: Security problem while authenticating: 
AMQ229031: Unable to validate user from 127.0.0.1:41314. Username: null; 
SSL certificate subject DN: CN=<myhost.mydomain>, OU=PUPPET, O=FOREMAN, ST=North Carolina, C=US
2024-09-23 11:01:01,301 [thread=Thread-9 (activemq-netty-threads)] [=, org=, csid=] 
WARN  org.apache.activemq.artemis.core.protocol.stomp - AMQ332069: Sent ERROR frame to STOMP 
client 127.0.0.1:41314: Security Error occurred: User name [null] or password is invalid

And /var/log/foreman/production.log shows errors about consumer ID not found:

2024-09-23T11:01:10 [E|app|0bdec618] RestClient::NotFound: Katello::Resources::Candlepin::Consumer: 404 Not Found 
{"displayMessage":"Consumer with ID(s) ee9363b1-0019-4bde-a468-e00f2790613d could not be found.",
"requestUuid":"e7c78eef-25b1-4f26-b1f7-38ecac53a8dc"} (GET /candlepin/consumers/ee9363b1-0019-4bde-a468-e00f2790613d) 0bdec618 | 
Body: {"displayMessage":"Consumer with ID(s) ee9363b1-0019-4bde-a468-e00f2790613d could not be found.",
"requestUuid":"e7c78eef-25b1-4f26-b1f7-38ecac53a8dc"}
 0bdec618 |
2024-09-23T11:01:10 [E|app|0bdec618] /usr/share/gems/gems/katello-4.13.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:323:in `block in find_host'

I’ve been unable to find anything specifying a user with ID e7c78eef-25b1-4f26-b1f7-38ecac53a8dc. Where would that be set? And why would it have stopped working when it was working previously?

Any suggestions on where to look to understand those errors?

User name [null] or password is invalid

Consumer with ID(s) ee9363b1-0019-4bde-a468-e00f2790613d could not be found.

I would focus on the user / authentication issues. The consumer is just a host that is no longer in Candlepin.

Thanks. I’ve been trying to get info on the user name null issue, but it’s not clear what user it’s referring to, or where I would set it. Unfortunately the only info I’ve been able to find on this error is hidden behind a Red Hat paywall (Error with STOMP client causing 'candlepin_events' to enter a failed state - Red Hat Customer Portal).

I do have a user name and password set in /etc/candlepin/candlepin.conf, so I’m confused about the error message.

I’ve managed to get candlepin_events to start by setting security-enabled to false in /etc/candlepin/broker.xml, but I’m not very comfortable with that. I’d rather find where the user authentication is actually failing, but I haven’t been able to figure that out from the log messages.

I can’t vouch for it but basically it says to stop services, extract something from /var/log/candlepin/error.log:

# echo "katelloUser=$(grep -o DN.* /var/log/candlepin/error.log | sort -u | cut -d' ' -f2-7)"

and then put that in /etc/tomcat/cert-users.properties. And then restart services. No idea if that will help but maybe something to try…

Thanks! That indeed fixed it, though I’m still confused about why that step was necessary. In /etc/tomcat/cert-users.properties, all I did was change

katelloUser=CN=myhost.mydomain, OU=PUPPET, O=FOREMAN, ST=Vermont, C=US
to:
katelloUser=CN=myhost.mydomain, OU=PUPPET, O=FOREMAN, ST=North Carolina, C=US

i.e. the state field was wrong in the original version. We’re using a custom cert from Let’s Encrypt everywhere, and I assume that Vermont (which is our location) was put there by foreman-installer. So why would Puppet only work when we manually changed that to North Carolina (which is, what, Puppet’s location?)?

It’s Red Hat’s location :slight_smile:

I think the certificate config in Foreman changed at some point, and it may be a leftover from a previous custom cert setup.