Problem:
Katello 4.12.1 server , after upgrade from Oracle Linux 8.9 to 8.10 , candlepin service failed , thanks.
$ hammer ping
database:
Status: ok
Server Response: Duration: 0ms
cache:
servers:
1) Status: ok
Server Response: Duration: 0ms
candlepin:
Status: FAIL
Server Response: Message: Failed to open TCP connection to localhost:23443 (Cannot assign requested address - connect(2) for “localhost” port 23443)
candlepin_auth:
Status: FAIL
Server Response: Message: Failed to open TCP connection to localhost:23443 (Cannot assign requested address - connect(2) for “localhost” port 23443)
candlepin_events:
Status: FAIL
message: Not running
Server Response: Duration: 0ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
pulp3:
Status: ok
Server Response: Duration: 443ms
pulp3_content:
Status: ok
Server Response: Duration: 53ms
foreman_tasks:
Status: ok
Server Response: Duration: 4ms
Foreman and Proxy versions:
$ rpm -q katello foreman
katello-4.12.1-1.el8.noarch
foreman-3.10.0-1.el8.noarch
Foreman and Proxy plugin versions:
Distribution and version:
$ cat /etc/oracle-release
Oracle Linux Server release 8.10
$ sestatus
SELinux status: disabled
$ uname -r
4.18.0-553.5.1.el8_10.x86_64
Other relevant data:
logs in /var/log/candlepin/candlepin.log:
2024-06-20 23:26:18,267 [thread=Thread-0 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to d
ecode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$843/0x00007f144099c938@1d8764f6 rejected from o
rg.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@155e551e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:387)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1355)
at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:211)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$843/0x00007f144099c938@1d8764f6 rejected from org.apache.activemq.art
emis.utils.ActiveMQThreadPoolExecutor@155e551e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1562)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:409)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:382)
... 8 common frames omitted
onFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1358)
at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:211)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-06-20 23:26:18,267 [thread=Thread-0 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN org.apache.activemq.artemis.utils.actors.OrderedExecutor - Faile
d to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@40b77338
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@40b77338
at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:219)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@66b10bcc rejected from org.apache.ac
tivemq.artemis.utils.ActiveMQThreadPoolExecutor@155e551e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1358)
at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:211)
... 6 common frames omitted
2024-06-20 23:26:18,502 [thread=Thread-11] [=, org=, csid=] INFO org.candlepin.messaging.impl.artemis.ArtemisContextListener - Embedded Artemis server stopped