Feedback for Foreman 3.11 & Katello 4.13

lumarel · July 26, 2024, 10:47am

Yeah it’s pretty weird
Though just so say I’m grabbing all that via the phone from remote right now I hope I don’t moss something.
Will do another complete run soon when I have access to the workstation, maybe I’m missing something.

In the localhost log there is only a message from the tomcat stop from like an hour before the last output.

The candlepin log has the following in it:

2024-07-26 12:07:32,127 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2024-07-26 12:07:32,138 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2024-07-26 12:07:32,144 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2024-07-26 12:07:32,257 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2024-07-26 12:07:32,289 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2024-07-26 12:07:32,307 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2024-07-26 12:07:32,608 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage

lumarel · July 26, 2024, 11:49am

So, I collected the following logs with the process:

All packages up to date, except candlepin and candlepin-selinux on 4.4.10
Foreman on 3.11.1 and Katello on 4.13.1
Everything running without issues right now
Doing the dnf update to update candlepin

The following messages might have occured while doing the update or right after.

==> /var/log/tomcat/catalina.2024-07-26.log <==
26-Jul-2024 13:37:31.799 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.reload Reloading context [/candlepin]
26-Jul-2024 13:37:31.799 INFO [Catalina-utility-1] org.apache.catalina.core.StandardContext.reload Reloading Context with name [/candlepin] has started
26-Jul-2024 13:37:32.341 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3d3835c2]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4c9a69cc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Jul-2024 13:37:32.342 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@409d58d3]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(1921267161<closed>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Jul-2024 13:37:32.342 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@e6e886e]) and a value of type [liquibase.SingletonScopeManager] (value [liquibase.SingletonScopeManager@11255466]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Jul-2024 13:37:32.342 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3d3835c2]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@3bac8789]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Jul-2024 13:37:32.350 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4147da62]) and a value of type [org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl] (value [OperationContextImpl [1592894089] [minimalStore=9223372036854775807, storeLineUp=1, stored=1, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Jul-2024 13:37:32.351 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3d3835c2]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@5c032a6c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Jul-2024 13:37:32.932 INFO [Finalizer] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [io.netty.util.internal.shaded.org.jctools.queues.IndexedQueueSizeUtil]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
	java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [io.netty.util.internal.shaded.org.jctools.queues.IndexedQueueSizeUtil]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
		at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1349)
		at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1337)
		at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1174)
		at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1141)
		at io.netty.util.internal.shaded.org.jctools.queues.ConcurrentCircularArrayQueue.isEmpty(ConcurrentCircularArrayQueue.java:71)
		at io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue.isEmpty(MpscArrayQueue.java:207)
		at io.netty.buffer.PoolThreadCache.checkCacheMayLeak(PoolThreadCache.java:236)
		at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:227)
		at io.netty.buffer.PoolThreadCache$FreeOnFinalize.finalize(PoolThreadCache.java:500)
		at java.base/java.lang.System$2.invokeFinalize(System.java:2301)
		at java.base/java.lang.ref.Finalizer.runFinalizer(Finalizer.java:88)
		at java.base/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:173)
26-Jul-2024 13:37:35.531 INFO [Catalina-utility-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:36:58,995 [thread=https-jsse-nio-127.0.0.1-23443-exec-10] [req=df8eaf9a-ff02-4df9-b5ea-a838589e2797, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/8cc7a859-c991-4bcd-b350-808d703178f1/certificates/serials
2024-07-26 13:36:59,033 [thread=https-jsse-nio-127.0.0.1-23443-exec-10] [req=df8eaf9a-ff02-4df9-b5ea-a838589e2797, org=LUMA, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=38
2024-07-26 13:36:59,075 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=5cae2812-cd5c-4d25-b1c5-fc8fe8b203fd, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/status
2024-07-26 13:36:59,076 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=5cae2812-cd5c-4d25-b1c5-fc8fe8b203fd, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=1
2024-07-26 13:36:59,117 [thread=https-jsse-nio-127.0.0.1-23443-exec-3] [req=b63962f3-1b36-4cfd-bf0d-c051d7cfc7cc, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/8cc7a859-c991-4bcd-b350-808d703178f1/accessible_content
2024-07-26 13:36:59,127 [thread=https-jsse-nio-127.0.0.1-23443-exec-3] [req=b63962f3-1b36-4cfd-bf0d-c051d7cfc7cc, org=LUMA, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=304, content-type="application/json", time=10
2024-07-26 13:36:59,169 [thread=https-jsse-nio-127.0.0.1-23443-exec-10] [req=60613b6a-ac9c-46b7-ae31-90fe09f70d07, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/8cc7a859-c991-4bcd-b350-808d703178f1/content_overrides
2024-07-26 13:36:59,178 [thread=https-jsse-nio-127.0.0.1-23443-exec-10] [req=60613b6a-ac9c-46b7-ae31-90fe09f70d07, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=9
2024-07-26 13:37:11,420 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=ee7bb45c-40c5-46b8-b6a5-0d909d5d3cf3, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/status
2024-07-26 13:37:11,423 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=ee7bb45c-40c5-46b8-b6a5-0d909d5d3cf3, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=3
2024-07-26 13:37:11,459 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=6e396a94-e2b1-40d4-a750-9d44bd35f1a2, org=, csid=55a250e7-9845-4bc1-9131-cf0e65d9e6bf] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/4ae71432-e8c9-4554-b06a-9d41a417bd0c
2024-07-26 13:37:11,499 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=6e396a94-e2b1-40d4-a750-9d44bd35f1a2, org=LUMA, csid=55a250e7-9845-4bc1-9131-cf0e65d9e6bf] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=40
2024-07-26 13:37:11,560 [thread=https-jsse-nio-127.0.0.1-23443-exec-1] [req=c9e7072a-407c-4188-9f9d-349dc8fb758d, org=, csid=c39f57dc-9b1e-4f3d-87eb-4d90f3622dca] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/4ae71432-e8c9-4554-b06a-9d41a417bd0c
2024-07-26 13:37:11,590 [thread=https-jsse-nio-127.0.0.1-23443-exec-1] [req=c9e7072a-407c-4188-9f9d-349dc8fb758d, org=LUMA, csid=c39f57dc-9b1e-4f3d-87eb-4d90f3622dca] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=30
2024-07-26 13:37:11,631 [thread=https-jsse-nio-127.0.0.1-23443-exec-8] [req=35c2ef02-46aa-462d-accd-14f1c2d0539d, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/4ae71432-e8c9-4554-b06a-9d41a417bd0c/compliance
2024-07-26 13:37:11,671 [thread=https-jsse-nio-127.0.0.1-23443-exec-8] [req=35c2ef02-46aa-462d-accd-14f1c2d0539d, org=LUMA, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=40
2024-07-26 13:37:11,728 [thread=https-jsse-nio-127.0.0.1-23443-exec-6] [req=feb719de-809e-4edc-85f0-4adf16579dab, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/4ae71432-e8c9-4554-b06a-9d41a417bd0c/certificates/serials
2024-07-26 13:37:11,761 [thread=https-jsse-nio-127.0.0.1-23443-exec-6] [req=feb719de-809e-4edc-85f0-4adf16579dab, org=LUMA, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=33
2024-07-26 13:37:11,827 [thread=https-jsse-nio-127.0.0.1-23443-exec-2] [req=3f1f3f9f-ab2e-4b36-9946-e76cc623759e, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/4ae71432-e8c9-4554-b06a-9d41a417bd0c/accessible_content
2024-07-26 13:37:11,845 [thread=https-jsse-nio-127.0.0.1-23443-exec-2] [req=3f1f3f9f-ab2e-4b36-9946-e76cc623759e, org=LUMA, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=304, content-type="application/json", time=18
2024-07-26 13:37:21,057 [thread=https-jsse-nio-127.0.0.1-23443-exec-9] [req=f24a3b6d-3947-4f62-9af0-e99233451181, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/82022984-0258-4d12-abfc-ea01414de566/certificates/serials
2024-07-26 13:37:21,089 [thread=https-jsse-nio-127.0.0.1-23443-exec-9] [req=f24a3b6d-3947-4f62-9af0-e99233451181, org=LUMA, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=32
2024-07-26 13:37:21,134 [thread=https-jsse-nio-127.0.0.1-23443-exec-5] [req=a39de66d-e71c-46f0-bbda-9720f412decf, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/status
2024-07-26 13:37:21,137 [thread=https-jsse-nio-127.0.0.1-23443-exec-5] [req=a39de66d-e71c-46f0-bbda-9720f412decf, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=3
2024-07-26 13:37:21,184 [thread=https-jsse-nio-127.0.0.1-23443-exec-3] [req=a12c5405-5c01-4bc3-abcb-7893d5eb3f67, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/82022984-0258-4d12-abfc-ea01414de566/accessible_content
2024-07-26 13:37:21,197 [thread=https-jsse-nio-127.0.0.1-23443-exec-3] [req=a12c5405-5c01-4bc3-abcb-7893d5eb3f67, org=LUMA, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=304, content-type="application/json", time=13
2024-07-26 13:37:21,265 [thread=https-jsse-nio-127.0.0.1-23443-exec-10] [req=ac531529-d6d0-44e4-9a81-45762c800dc8, org=, csid=5a7bbbcd-28ad-451b-8dd0-34b2da25c627] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/82022984-0258-4d12-abfc-ea01414de566
2024-07-26 13:37:21,284 [thread=https-jsse-nio-127.0.0.1-23443-exec-10] [req=ac531529-d6d0-44e4-9a81-45762c800dc8, org=LUMA, csid=5a7bbbcd-28ad-451b-8dd0-34b2da25c627] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=19
2024-07-26 13:37:21,335 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=69fa9576-9598-4680-a547-769dc961b5e7, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/82022984-0258-4d12-abfc-ea01414de566/content_overrides
2024-07-26 13:37:21,346 [thread=https-jsse-nio-127.0.0.1-23443-exec-7] [req=69fa9576-9598-4680-a547-769dc961b5e7, org=, csid=] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=11
2024-07-26 13:37:23,377 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=4fb2650b-192a-43df-94a4-6b1f883d2674, org=, csid=5d328f9a-2266-4bbe-83f3-8a4d10dce8ac] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/82022984-0258-4d12-abfc-ea01414de566
2024-07-26 13:37:23,415 [thread=https-jsse-nio-127.0.0.1-23443-exec-4] [req=4fb2650b-192a-43df-94a4-6b1f883d2674, org=LUMA, csid=5d328f9a-2266-4bbe-83f3-8a4d10dce8ac] INFO  org.candlepin.servlet.filter.logging.LoggingFilter - Response: status=200, content-type="application/json", time=38
2024-07-26 13:37:31,800 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Shutting down job manager
2024-07-26 13:37:32,160 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Job manager shut down
2024-07-26 13:37:32,198 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214013: Failed to decode packet
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@241433a1 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,198 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214013: Failed to decode packet
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@241433a1 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,200 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to decode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@241433a1 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:397)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@241433a1 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	... 8 common frames omitted
2024-07-26 13:37:32,201 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224027: Failed to write to handler on invm connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@7fba1808
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@3a52b17b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-07-26 13:37:32,202 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@7fba1808
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@7fba1808
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:231)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@3a52b17b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	... 6 common frames omitted

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,200 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to decode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@241433a1 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:397)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@241433a1 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	... 8 common frames omitted
2024-07-26 13:37:32,201 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224027: Failed to write to handler on invm connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@7fba1808
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@3a52b17b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-07-26 13:37:32,202 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@7fba1808
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@7fba1808
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:231)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@3a52b17b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	... 6 common frames omitted

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,208 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214013: Failed to decode packet
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@74ce81dc rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,208 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214013: Failed to decode packet
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@74ce81dc rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,208 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to decode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@74ce81dc rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:397)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@74ce81dc rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	... 8 common frames omitted

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,208 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to decode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@74ce81dc rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:397)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@74ce81dc rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	... 8 common frames omitted

lumarel · July 26, 2024, 11:50am

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,208 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224027: Failed to write to handler on invm connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@5355c76
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@575ed65 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,208 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224027: Failed to write to handler on invm connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@5355c76
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@575ed65 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,209 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@5355c76
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@5355c76
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:231)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@575ed65 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	... 6 common frames omitted

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,209 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@5355c76
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@5355c76
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:231)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@575ed65 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	... 6 common frames omitted

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,209 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214013: Failed to decode packet
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@2574edaf rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,209 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214013: Failed to decode packet
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@2574edaf rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,210 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to decode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@2574edaf rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:397)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@2574edaf rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	... 8 common frames omitted

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,210 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.client - AMQ214031: Failed to decode buffer, disconnect immediately.
java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@2574edaf rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:397)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1365)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$1124/0x00007fedb4aa6b08@2574edaf rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:199)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:187)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:72)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.nodeDisconnected(ClientSessionFactoryImpl.java:1572)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handleDisconnect(ActiveMQClientProtocolManager.java:505)
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:484)
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:835)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:419)
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:392)
	... 8 common frames omitted

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,210 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224027: Failed to write to handler on invm connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@f7932c
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@2ec1f4b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-07-26 13:37:32,210 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@f7932c
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@f7932c
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:231)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@2ec1f4b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	... 6 common frames omitted

==> /var/log/candlepin/error.log <==
2024-07-26 13:37:32,210 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224027: Failed to write to handler on invm connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@f7932c
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@2ec1f4b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-07-26 13:37:32,210 [thread=Thread-6 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@f7932c
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@f7932c
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:231)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@2ec1f4b rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@59898202[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1368)
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:223)
	... 6 common frames omitted


==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:37:32,337 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.messaging.impl.artemis.ArtemisContextListener - Embedded Artemis server stopped
2024-07-26 13:37:35,708 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2024-07-26 13:37:35,709 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2024-07-26 13:37:35,713 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2024-07-26 13:37:35,752 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2024-07-26 13:37:35,757 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2024-07-26 13:37:35,763 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2024-07-26 13:37:35,852 [thread=Catalina-utility-1] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage

==> /var/log/messages <==
Jul 26 13:37:27 r8-foreman-prod kernel: SELinux:  Converting 489 SID table entries...
Jul 26 13:37:27 r8-foreman-prod kernel: SELinux:  policy capability network_peer_controls=1
Jul 26 13:37:27 r8-foreman-prod kernel: SELinux:  policy capability open_perms=1
Jul 26 13:37:27 r8-foreman-prod kernel: SELinux:  policy capability extended_socket_class=1
Jul 26 13:37:27 r8-foreman-prod kernel: SELinux:  policy capability always_check_network=0
Jul 26 13:37:27 r8-foreman-prod kernel: SELinux:  policy capability cgroup_seclabel=1
Jul 26 13:37:27 r8-foreman-prod kernel: SELinux:  policy capability nnp_nosuid_transition=1
Jul 26 13:37:27 r8-foreman-prod dbus-daemon[961]: [system] Reloaded configuration
Jul 26 13:37:29 r8-foreman-prod kernel: SELinux:  Converting 489 SID table entries...
Jul 26 13:37:29 r8-foreman-prod kernel: SELinux:  policy capability network_peer_controls=1
Jul 26 13:37:29 r8-foreman-prod kernel: SELinux:  policy capability open_perms=1
Jul 26 13:37:29 r8-foreman-prod kernel: SELinux:  policy capability extended_socket_class=1
Jul 26 13:37:29 r8-foreman-prod kernel: SELinux:  policy capability always_check_network=0
Jul 26 13:37:29 r8-foreman-prod kernel: SELinux:  policy capability cgroup_seclabel=1
Jul 26 13:37:29 r8-foreman-prod kernel: SELinux:  policy capability nnp_nosuid_transition=1
Jul 26 13:37:29 r8-foreman-prod dbus-daemon[961]: [system] Reloaded configuration
Jul 26 13:37:31 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:31.799 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.reload Reloading context [/candlepin]
Jul 26 13:37:31 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:31.799 INFO [Catalina-utility-1] org.apache.catalina.core.StandardContext.reload Reloading Context with name [/candlepin] has started
Jul 26 13:37:32 r8-foreman-prod foreman[127427]: warning: broker sent EOF, and connection not reliable
Jul 26 13:37:32 r8-foreman-prod foreman[127427]: #<Thread:0x00005595f6c314a0 /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 26 13:37:32 r8-foreman-prod foreman[127427]: /usr/share/gems/gems/stomp-1.4.10/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 26 13:37:32 r8-foreman-prod foreman[127427]: #011from /usr/share/gems/gems/stomp-1.4.10/lib/client/utils.rb:194:in `loop'
Jul 26 13:37:32 r8-foreman-prod foreman[127427]: #011from /usr/share/gems/gems/stomp-1.4.10/lib/client/utils.rb:194:in `block in start_listeners'
Jul 26 13:37:32 r8-foreman-prod foreman[127427]: #011from /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 26 13:37:32 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:32.341 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3d3835c2]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4c9a69cc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 26 13:37:32 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:32.342 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@409d58d3]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(1921267161<closed>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 26 13:37:32 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:32.342 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@e6e886e]) and a value of type [liquibase.SingletonScopeManager] (value [liquibase.SingletonScopeManager@11255466]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 26 13:37:32 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:32.342 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3d3835c2]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@3bac8789]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 26 13:37:32 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:32.350 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4147da62]) and a value of type [org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl] (value [OperationContextImpl [1592894089] [minimalStore=9223372036854775807, storeLineUp=1, stored=1, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 26 13:37:32 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:32.351 SEVERE [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3d3835c2]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@5c032a6c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 26 13:37:32 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:32.932 INFO [Finalizer] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [io.netty.util.internal.shaded.org.jctools.queues.IndexedQueueSizeUtil]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [io.netty.util.internal.shaded.org.jctools.queues.IndexedQueueSizeUtil]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1349)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1337)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1174)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1141)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at io.netty.util.internal.shaded.org.jctools.queues.ConcurrentCircularArrayQueue.isEmpty(ConcurrentCircularArrayQueue.java:71)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue.isEmpty(MpscArrayQueue.java:207)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at io.netty.buffer.PoolThreadCache.checkCacheMayLeak(PoolThreadCache.java:236)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:227)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at io.netty.buffer.PoolThreadCache$FreeOnFinalize.finalize(PoolThreadCache.java:500)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at java.base/java.lang.System$2.invokeFinalize(System.java:2301)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at java.base/java.lang.ref.Finalizer.runFinalizer(Finalizer.java:88)
Jul 26 13:37:32 r8-foreman-prod server[126880]: #011#011at java.base/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:173)
Jul 26 13:37:35 r8-foreman-prod server[126880]: 26-Jul-2024 13:37:35.531 INFO [Catalina-utility-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

==> /var/log/foreman/production.log <==
2024-07-26T13:37:40 [E|app|7be03b87] Error occurred while starting Katello::CandlepinEventListener
2024-07-26T13:37:40 [E|app|7be03b87] Connection refused - connect(2) for "localhost" port 61613
2024-07-26T13:37:40 [E|app|7be03b87] /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in `initialize'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in `open'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in `block in open_ssl_socket'
 7be03b87 | /usr/share/ruby/timeout.rb:78:in `timeout'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:460:in `open_ssl_socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:520:in `open_socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:116:in `block in socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:109:in `synchronize'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:109:in `socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/connection.rb:173:in `initialize'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:134:in `new'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:134:in `create_connection'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:101:in `block in initialize'
 7be03b87 | /usr/share/ruby/timeout.rb:95:in `block in timeout'
 7be03b87 | /usr/share/ruby/timeout.rb:33:in `block in catch'
 7be03b87 | /usr/share/ruby/timeout.rb:33:in `catch'
 7be03b87 | /usr/share/ruby/timeout.rb:33:in `catch'
 7be03b87 | /usr/share/ruby/timeout.rb:110:in `timeout'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:99:in `initialize'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:69:in `new'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:69:in `client'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:43:in `subscribe'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/services/katello/candlepin_event_listener.rb:34:in `run'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:40:in `block in check_services'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `each'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `check_services'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:16:in `block (2 levels) in start'
 7be03b87 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:87:in `wrap'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:15:in `block in start'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `loop'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `start'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:84:in `block (2 levels) in start_monitor_thread'
 7be03b87 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:83:in `block in start_monitor_thread'
 7be03b87 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

This I caught fairly everything now.

lumarel · July 26, 2024, 12:03pm

And I let foreman-installer run for a while longer now, it actually completes.

I think Tomcat restarted during that process:

==> /var/log/tomcat/catalina.2024-07-26.log <==
26-Jul-2024 13:54:16.223 INFO [Thread-10] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
26-Jul-2024 13:54:16.224 INFO [Thread-10] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
26-Jul-2024 13:55:46.946 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
26-Jul-2024 13:55:47.306 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
26-Jul-2024 13:55:47.583 WARNING [main] org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the [ciphers] attribute in a manner consistent with the latest OpenSSL development branch. Some of the specified [ciphers] are not supported by the configured SSL engine for this connector (which may use JSSE or an older OpenSSL version) and have been skipped: [[TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256]]
26-Jul-2024 13:55:48.014 INFO [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-jsse-nio-127.0.0.1-23443], TLS virtual host [_default_], certificate type [UNDEFINED] configured from keystore [/etc/candlepin/certs/keystore] using alias [tomcat] with trust store [/etc/candlepin/certs/truststore]
26-Jul-2024 13:55:48.022 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1326] milliseconds
26-Jul-2024 13:55:48.060 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
26-Jul-2024 13:55:48.060 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
26-Jul-2024 13:55:48.067 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
26-Jul-2024 13:55:52.315 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

==> /var/log/candlepin/candlepin.log <==
2024-07-26 13:55:52,599 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2024-07-26 13:55:52,601 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2024-07-26 13:55:52,607 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2024-07-26 13:55:52,669 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2024-07-26 13:55:52,678 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2024-07-26 13:55:52,686 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2024-07-26 13:55:52,835 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage

But yeah the recurring error in Katello is still there:

==> /var/log/foreman/production.log <==
2024-07-26T14:02:56 [E|app|7be03b87] Error occurred while starting Katello::CandlepinEventListener
2024-07-26T14:02:56 [E|app|7be03b87] Connection refused - connect(2) for "localhost" port 61613
2024-07-26T14:02:56 [E|app|7be03b87] /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in `initialize'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in `open'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in `block in open_ssl_socket'
 7be03b87 | /usr/share/ruby/timeout.rb:78:in `timeout'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:460:in `open_ssl_socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:520:in `open_socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:116:in `block in socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:109:in `synchronize'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:109:in `socket'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/connection.rb:173:in `initialize'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:134:in `new'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:134:in `create_connection'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:101:in `block in initialize'
 7be03b87 | /usr/share/ruby/timeout.rb:95:in `block in timeout'
 7be03b87 | /usr/share/ruby/timeout.rb:33:in `block in catch'
 7be03b87 | /usr/share/ruby/timeout.rb:33:in `catch'
 7be03b87 | /usr/share/ruby/timeout.rb:33:in `catch'
 7be03b87 | /usr/share/ruby/timeout.rb:110:in `timeout'
 7be03b87 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:99:in `initialize'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:69:in `new'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:69:in `client'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/messaging/stomp_connection.rb:43:in `subscribe'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/services/katello/candlepin_event_listener.rb:34:in `run'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:40:in `block in check_services'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `each'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:30:in `check_services'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:16:in `block (2 levels) in start'
 7be03b87 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:87:in `wrap'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:15:in `block in start'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `loop'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/monitor.rb:14:in `start'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:84:in `block (2 levels) in start_monitor_thread'
 7be03b87 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
 7be03b87 | /usr/share/gems/gems/katello-4.13.1/app/lib/katello/event_daemon/runner.rb:83:in `block in start_monitor_thread'
 7be03b87 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

nikos_moum · July 26, 2024, 1:12pm

All the RejectedExecutionException errors should be benign; they happen when tomcat is being shut down.
Both the candlepin.log and the catalina.log seem to end prematurely. I would expect further messages for both of those.
Can you check the /var/log/tomcat/localhost.log (not catalina)

lumarel · July 26, 2024, 4:26pm

Hmm okay, same as earlier, there is no new log in localhost.log since a long time, the latest message is still the one from hours before.

lumarel · July 26, 2024, 4:33pm

To add on that, I have been watching the whole /var/log/tomcat/*.log until now, all messages seen from that directory until now (even until now), are in the last messages above. Same for /var/log/candlepin/*.log
As well as foreman-tail, where I picked out the important messages, because there are also a few clients talking.

lumarel · July 26, 2024, 4:59pm

tbh, it looks like Tomcat fully starts up without error and also starts at least part of the candlepin application, but just halts somewhere then.
Also took a look about the port 61613, looks like that never comes up, at least also now after the hours there is still nothing listening there.

GrinGrin · July 26, 2024, 8:53pm

Hi all !

I found this post while trying to figure out why I can’t upgrade to Foreman 3.11.1/Katello 4.13.1 on a RockyLinux 8 server.

I was able to migrate another server to Foreman 3.11.1/Katello 4.13 .0 few days before but here I was stuck with a lot of errors during foreman-installer upgrade phase (errors 500, connection to database lost, huge stack traces in /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb …) and the whole process was very very slow until it failed

Will digging into the katello.log upgrade file, I saw a lot of errors about Katello::Errors::CandlepinNotRunning and Connection refused - connect(2) for "localhost" port 61613 . Netstat confirm the port is not open unlike the other server that have the port 61613 hold by the tomcat process.

I then checked catalina log and did not see any error at all, just a few lines at each startup/restart of tomcat:

26-Jul-2024 21:51:21.110 WARNING [main] org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the [ciphers] attribute in a manner consistent with the latest OpenSSL development branch. Some of the specified [ciphers] are not supported by the configured SSL engine for this connector (which may use JSSE or an older OpenSSL version) and have been skipped: [[TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256]]
26-Jul-2024 21:51:22.272 INFO [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-jsse-nio-127.0.0.1-23443], TLS virtual host [_default_], certificate type [UNDEFINED] configured from keystore [/etc/candlepin/certs/keystore] using alias [tomcat] with trust store [/etc/candlepin/certs/truststore]
26-Jul-2024 21:51:22.304 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [3820] milliseconds
26-Jul-2024 21:51:22.432 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
26-Jul-2024 21:51:22.432 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
26-Jul-2024 21:51:22.462 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
26-Jul-2024 21:51:28.104 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
26-Jul-2024 21:59:59.650 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]

On the other server, after the same lines there was a bunch of liquibase logs that were not present here…

I checked candlepin packages on both servers when I see @evgeni talking about the 4.4.10 version and @lumarel stating that it failed again with 4.4.12:

faulty server has candlepin-4.4.12-1.el8.noarch / candlepin-selinux-4.4.12-1.el8.noarch
working server is still on candlepin-selinux-4.4.10-1.el8.noarch / candlepin-4.4.10-1.el8.noarch

After downgrading candlepin from 4.4.12 to 4.4 .10 (dnf downgrade candlepin candlepin-selinux) the tomcat service started correctly and produced the liquibase logs:

26-Jul-2024 22:06:51.318 WARNING [main] org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the [ciphers] attribute in a manner consistent with the latest OpenSSL development branch. Some of the specified [ciphers] are not supported by the configured SSL engine for this connector (which may use JSSE or an older OpenSSL version) and have been skipped: [[TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256]]
26-Jul-2024 22:06:52.536 INFO [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-jsse-nio-127.0.0.1-23443], TLS virtual host [_default_], certificate type [UNDEFINED] configured from keystore [/etc/candlepin/certs/keystore] using alias [tomcat] with trust store [/etc/candlepin/certs/truststore]
26-Jul-2024 22:06:52.567 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [4054] milliseconds
26-Jul-2024 22:06:52.708 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
26-Jul-2024 22:06:52.709 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
26-Jul-2024 22:06:52.794 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
26-Jul-2024 22:06:58.788 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
26-Jul-2024 22:07:02.307 INFO [main] liquibase.database.null Set default schema name to public
26-Jul-2024 22:07:02.476 INFO [main] liquibase.changelog.null Reading from public.databasechangelog
26-Jul-2024 22:07:04.710 INFO [main] liquibase.lockservice.null Successfully acquired change log lock
26-Jul-2024 22:07:04.712 INFO [main] liquibase.command.null Using deploymentId: 2024424712
26-Jul-2024 22:07:04.715 INFO [main] liquibase.changelog.null Reading from public.databasechangelog
26-Jul-2024 22:07:04.783 INFO [main] liquibase.changelog.null Columns anonymous(boolean),claimed(boolean) added to cp_owner
26-Jul-2024 22:07:04.787 INFO [main] liquibase.changelog.null ChangeSet db/changelog/20230725171517-add-anonymous-claimed_column.xml::20230725171517-1::sbakaj ran successfully in 13ms
26-Jul-2024 22:07:04.811 INFO [main] liquibase.snapshot.null Creating snapshot
26-Jul-2024 22:07:04.867 INFO [main] liquibase.changelog.null Table cp_anonymous_cloud_consumers created
[... redacted for readability ...]
26-Jul-2024 22:07:07.922 INFO [main] liquibase.util.null UPDATE SUMMARY
26-Jul-2024 22:07:07.923 INFO [main] liquibase.util.null Run:                         33
26-Jul-2024 22:07:07.923 INFO [main] liquibase.util.null Previously run:             611
26-Jul-2024 22:07:07.923 INFO [main] liquibase.util.null Filtered out:               207
26-Jul-2024 22:07:07.923 INFO [main] liquibase.util.null -------------------------------
26-Jul-2024 22:07:07.923 INFO [main] liquibase.util.null Total change sets:          851
26-Jul-2024 22:07:07.923 INFO [main] liquibase.util.null FILTERED CHANGE SETS SUMMARY
26-Jul-2024 22:07:07.923 INFO [main] liquibase.util.null DBMS mismatch:              207
26-Jul-2024 22:07:07.931 INFO [main] liquibase.util.null Update summary generated
26-Jul-2024 22:07:07.938 INFO [main] liquibase.command.null Update command completed successfully.
26-Jul-2024 22:07:07.942 INFO [main] liquibase.lockservice.null Successfully released change log lock
26-Jul-2024 22:07:07.943 INFO [main] liquibase.command.null Command execution complete

Also, the production.log file was not flooded with errors anymore !
I then restarted foreman-installer to finally complete correctly the upgrade and now my Foreman instance seems fine and running as intended…
I will do some checks to see if it is stable enough or if I need to revert the VM snapshot to get back to Foreman 3.10/Katello 4.12.

Hope this will help find out what happened between 4.4.10 and 4.4.12 !

Best Regards,
Nicolas.

gvde · July 28, 2024, 3:16pm

I can confirm that candlepin-4.4.12 causes serious issues with the whole foreman/katello system including issues during upgrade to 3.11/4.13. We have ran into the same issue in Upgrade from 3.10.0/4.12.1 to 3.11.1/4.13.1 makes foreman unusable. Downgrading candlepin and restarting tomcat seems to solve the problems.

nikos_moum · July 29, 2024, 10:31am

I have created 2300323 – candlepin-4.4.12-1 DB upgrade gets stuck to track this issue

nikos_moum · July 29, 2024, 1:17pm

Hi folks, we’re having trouble reproducing the issue. We believe the state of the DB has something to do with the issue. Can someone provide a DB dump to help us reproduce?

lumarel · July 29, 2024, 1:24pm

If only the dump of the candlepin db is necessary, I can send that to you right away (sent)

nikos_moum · July 29, 2024, 3:52pm

Even with the db dump, we are unable to reproduce the issue locally. So I’m at a loss for what to do. I noticed one of the other people that reproduced saw a change in candlepin.conf when upgrading (here: Foreman 3.11 upgrade - #2 by lumarel). Could you try to upgrade again, and check for any abnormalities/changes in candlepin.conf?

gvde · July 29, 2024, 5:19pm

I have checked on my upgraded snapshot. candlepin.conf looks fine to me. It’s identical except for a few blank lines and this addition:

> candlepin.db.database_manage_on_startup=Manage

I have got everything stopped on the snapshot except postgresql and then started tomcat.service. Candlepin is still stuck, doesn’t respond on port 23443. The last line logged from tomcat.service is

org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs

(which is always there, so not a problem) With the previous candlepin version, it continued with

liquibase.database.null Set default schema name to public

The connection to the database is active, but according to pg_stat_activity, idle in transaction with select setting from pg_settings where name = 'edb_redwood_date'. I think that means that is the last query executed and now it’s waiting for the next but nothings coming.

After removing the line

candlepin.db.database_manage_on_startup=Manage

from candlepin.conf, candlepin starts again and connections to port 23443 are possible again. I don’t have time to test further but at the moment it looks to me as if that added line could be the culprit…

jeremylenz · July 29, 2024, 5:29pm

gvde:

After removing the line
candlepin.db.database_manage_on_startup=Manage
from candlepin.conf, candlepin starts again and connections to port 23443 are possible again. I don’t have time to test further but at the moment it looks to me as if that added line could be the culprit…

Interesting. This was added here: Support Candlepin 4.4 by ehelms · Pull Request #259 · theforeman/puppet-candlepin · GitHub

lumarel · July 29, 2024, 6:52pm

I too didn’t really see any differences in the /etc/candlepin and /var/lib/candlepin directories,
so seems like doing the candlepin 4.4.10 → 4.4.12 update doesn’t touch the /etc/candlepin/candlepin.conf file.

But removing the line has definitely a positive effect, don’t see any errors after commenting that out.
Looks like 4.4.10 treats that line differently than 4.4.12.

Ceiu · July 29, 2024, 6:58pm

gvde:

After removing the line
candlepin.db.database_manage_on_startup=Manage
from candlepin.conf, candlepin starts again and connections to port 23443 are possible again. I don’t have time to test further but at the moment it looks to me as if that added line could be the culprit…

This definitely points us in a specific direction, and from the logs we’ve seen while digging into it a bit here I have a good guess as to what’s going wrong, but I do want to clarify what this particular configuration is:

This controls, as the name implies, if or how Candlepin will manage its database schema. When set to “manage”, “halt”, or “report”, Candlepin invokes our schema management library (Liquibase at the moment) to check if there are changesets present that haven’t been run, and, depending on that setting, run the change sets, halt Candlepin with an error message, or issue a warning and continue on.

If set to “None”, or the setting is removed entirely as is the case here, Candlepin will skip all of that and just assume the DB schema is being managed externally and go on its way without ever invoking Liquibase at all. This is fine for a very short-term workaround, but as soon as we make backward-incompatible schema changes, things will get very crashy, very quickly.

Now, given what we’ve seen from logs shared by some of y’all hitting this and the fact that shutting off Liquibase checks and management gets around the issue, we can make a pretty strong assertion that something is going sideways during the step where Liquibase checks what changes have been run.

One thing that might be helpful here is manually invoking the Liquibase upgrade and seeing if that kicks out any error that may be getting suppressed or otherwise lost when Candlepin is invoking it. I’ll work with our Katello friends to see if/how we can go about doing that from a proper Foreman environment and update the thread accordingly.

Ceiu · July 29, 2024, 7:09pm

Sounds like the old cpdb command is still around to manually invoke the schema update. Those of you hitting this issue, try running the following command and see if it halts/hangs in the same way or, better yet, errors out:

sudo /usr/share/candlepin/cpdb --update-schema

lumarel · July 29, 2024, 7:10pm

Thanks for the explenation!
With that knowledge, I tried to set it to report, but looks like it also gets stuck there,
definitely seeing the org.candlepin.database.MigrationManager - Liquibase startup management set to Report line though in the candlepin log. (same with Halt)
Though it’s the last message that pops up there.
When setting it to None it continues, as when completely removed