Candlepin service failed after upgrade to Oracle Linux 8.10

solar2k · June 21, 2024, 12:07am

Problem:
Katello 4.12.1 server , after upgrade from Oracle Linux 8.9 to 8.10 , candlepin service failed , thanks.

$ hammer ping
database:
Status: ok
Server Response: Duration: 0ms
cache:
servers:
1) Status: ok
Server Response: Duration: 0ms
candlepin:
Status: FAIL
Server Response: Message: Failed to open TCP connection to localhost:23443 (Cannot assign requested address - connect(2) for “localhost” port 23443)
candlepin_auth:
Status: FAIL
Server Response: Message: Failed to open TCP connection to localhost:23443 (Cannot assign requested address - connect(2) for “localhost” port 23443)
candlepin_events:
Status: FAIL
message: Not running
Server Response: Duration: 0ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
pulp3:
Status: ok
Server Response: Duration: 443ms
pulp3_content:
Status: ok
Server Response: Duration: 53ms
foreman_tasks:
Status: ok
Server Response: Duration: 4ms

Foreman and Proxy versions:

$ rpm -q katello foreman
katello-4.12.1-1.el8.noarch
foreman-3.10.0-1.el8.noarch

Foreman and Proxy plugin versions:

Distribution and version:

$ cat /etc/oracle-release
Oracle Linux Server release 8.10
$ sestatus
SELinux status: disabled
$ uname -r
4.18.0-553.5.1.el8_10.x86_64

Other relevant data:
logs in /var/log/candlepin/candlepin.log:

2024-06-20 23:26:18,267 [thread=Thread-0 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to d
ecode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$843/0x00007f144099c938@1d8764f6 rejected from o
rg.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@155e551e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:387)
        at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1355)
        at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:211)
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$843/0x00007f144099c938@1d8764f6 rejected from org.apache.activemq.art
emis.utils.ActiveMQThreadPoolExecutor@155e551e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
        at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
        at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
        at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
        at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1562)
        at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
        at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:409)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:382)
        ... 8 common frames omitted
        onFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1358)
                at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:211)
                at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
                at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
                at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
                at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
        2024-06-20 23:26:18,267 [thread=Thread-0 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Faile
        d to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@40b77338
        java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@40b77338
                at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:219)
                at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
                at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
                at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
                at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
        Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@66b10bcc rejected from org.apache.ac
        tivemq.artemis.utils.ActiveMQThreadPoolExecutor@155e551e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
                at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
                at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
                at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
                at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1358)
                at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:211)
                ... 6 common frames omitted
        2024-06-20 23:26:18,502 [thread=Thread-11] [=, org=, csid=] INFO  org.candlepin.messaging.impl.artemis.ArtemisContextListener - Embedded Artemis server stopped

evgeni · June 21, 2024, 7:00am

Hey,

this clearly shouldn’t happen, so let’s see if we can diagnose this further. Can you get me the outputs of the following commands?

rpm -q candlepin tomcat
systemctl status tomcat
curl -kv https://localhost:23443/candlepin/status/
curl -kv https://127.0.0.1:23443/candlepin/status/ (yes, really, once with hostname, once with ip, please)

“Cannot assign requested address” seems to indicate that the Ruby process can’t create a socket, even before the actual connection attempt.
Do you have any special IPv6 configuration or anything?

lbss · June 28, 2024, 10:02am

I had the same after 8.10 upgrade on Rocky. I found that the tomcat service was no longer enabled. Try running a “foreman-maintain service list” and see if tomcat is showing as disabled. Then run foreman-maintain service enable and its should get re-enabled. This solved the issue for me.

lbss · June 28, 2024, 11:11am

See other issue here which I also had to apply candlepin fix. Candlepin does not support JSS versions less than 4.4? - Support - TheForeman