Foreman 3.1 + Katello 4.3.0: Candlepin is not starting up

Problem:

This issue is on Centos 7.9, a new install on a new VM.

The installation completes without any errors, tomcat and other services are running, but candlepin is not starting up. I can visit the URL of the server in a browser but it only displays an error informing me that candlepin is not running

Logs below (I have omitted repeat entries), please let me know if you require more information or logs. I’m not sure where to start as I am new to Katello and Foreman.

localhost.22-02.01.log

Feb 01, 2022 10:32:01 AM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.candlepin.guice.CandlepinContextListener
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.io.IOException: Could not read key
  at org.candlepin.pki.CertificateReader.<init>(CertificateReader.java:50)
  at org.candlepin.guice.CandlepinModule.bindPki(CandlepinModule.java:303)
  while locating org.candlepin.pki.CertificateReader
    for the 1st parameter of org.candlepin.pki.impl.JSSPKIUtility.<init>(JSSPKIUtility.java:101)
  while locating org.candlepin.pki.impl.JSSPKIUtility
  at org.candlepin.guice.CandlepinModule.bindPki(CandlepinModule.java:302)
  while locating org.candlepin.pki.PKIUtility
Caused by: java.io.IOException: Could not read key
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader$PKCS8EncryptedPrivateKeyPemParser.decode(ProviderBasedPrivateKeyReader.java:210)
	at org.candlepin.pki.PrivateKeyReader$PrivateKeyPemParser.decode(PrivateKeyReader.java:47)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.readPem(ProviderBasedPrivateKeyReader.java:156)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.read(ProviderBasedPrivateKeyReader.java:82)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.read(ProviderBasedPrivateKeyReader.java:60)
	at org.candlepin.pki.CertificateReader.readPrivateKey(CertificateReader.java:78)
	at org.candlepin.pki.CertificateReader.<init>(CertificateReader.java:58)
	at org.candlepin.pki.CertificateReader$$FastClassByGuice$$d3225fea.newInstance(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:56)
	at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5127)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5643)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
	at java.base/javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122)
	at java.base/javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader$PKCS8EncryptedPrivateKeyPemParser.decode(ProviderBasedPrivateKeyReader.java:203)
	... 42 more

2) Error injecting constructor, java.io.IOException: Could not read key
  at org.candlepin.pki.CertificateReader.<init>(CertificateReader.java:50)
  at org.candlepin.guice.CandlepinModule.bindPki(CandlepinModule.java:303)
  while locating org.candlepin.pki.CertificateReader
Caused by: java.io.IOException: Could not read key
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader$PKCS8EncryptedPrivateKeyPemParser.decode(ProviderBasedPrivateKeyReader.java:210)
	at org.candlepin.pki.PrivateKeyReader$PrivateKeyPemParser.decode(PrivateKeyReader.java:47)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.readPem(ProviderBasedPrivateKeyReader.java:156)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.read(ProviderBasedPrivateKeyReader.java:82)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.read(ProviderBasedPrivateKeyReader.java:60)
	at org.candlepin.pki.CertificateReader.readPrivateKey(CertificateReader.java:78)
	at org.candlepin.pki.CertificateReader.<init>(CertificateReader.java:58)
	at org.candlepin.pki.CertificateReader$$FastClassByGuice$$d3225fea.newInstance(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:56)
	at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5127)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5643)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
	at java.base/javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122)
	at java.base/javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader$PKCS8EncryptedPrivateKeyPemParser.decode(ProviderBasedPrivateKeyReader.java:203)
	... 33 more

3) Error injecting constructor, java.io.IOException: Could not read key
  at org.candlepin.pki.CertificateReader.<init>(CertificateReader.java:50)
  at org.candlepin.guice.CandlepinModule.bindPki(CandlepinModule.java:303)
  while locating org.candlepin.pki.CertificateReader
    for the 1st parameter of org.candlepin.pki.impl.JSSPKIUtility.<init>(JSSPKIUtility.java:101)
  while locating org.candlepin.pki.impl.JSSPKIUtility
  at org.candlepin.guice.CandlepinModule.bindPki(CandlepinModule.java:302)
  while locating org.candlepin.pki.PKIUtility
    for the 1st parameter of org.candlepin.model.KeyPairCurator.<init>(KeyPairCurator.java:36)
  at org.candlepin.model.KeyPairCurator.class(KeyPairCurator.java:36)
  while locating org.candlepin.model.KeyPairCurator
Caused by: java.io.IOException: Could not read key
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader$PKCS8EncryptedPrivateKeyPemParser.decode(ProviderBasedPrivateKeyReader.java:210)
	at org.candlepin.pki.PrivateKeyReader$PrivateKeyPemParser.decode(PrivateKeyReader.java:47)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.readPem(ProviderBasedPrivateKeyReader.java:156)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.read(ProviderBasedPrivateKeyReader.java:82)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader.read(ProviderBasedPrivateKeyReader.java:60)
	at org.candlepin.pki.CertificateReader.readPrivateKey(CertificateReader.java:78)
	at org.candlepin.pki.CertificateReader.<init>(CertificateReader.java:58)
	at org.candlepin.pki.CertificateReader$$FastClassByGuice$$d3225fea.newInstance(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:56)
	at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5127)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5643)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
	at java.base/javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122)
	at java.base/javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168)
	at org.candlepin.pki.impl.ProviderBasedPrivateKeyReader$PKCS8EncryptedPrivateKeyPemParser.decode(ProviderBasedPrivateKeyReader.java:203)
	... 50 more

4) Error injecting constructor, java.io.IOException: Could not read key
  at org.candlepin.pki.CertificateReader.<init>(CertificateReader.java:50)
  at org.candlepin.guice.CandlepinModule.bindPki(CandlepinModule.java:303)
  while locating org.candlepin.pki.CertificateReader
    for the 1st parameter of org.candlepin.pki.impl.JSSPKIUtility.<init>(JSSPKIUtility.java:101)
  while locating org.candlepin.pki.impl.JSSPKIUtility
  at org.candlepin.guice.CandlepinModule.bindPki(CandlepinModule.java:302)
  while locating org.candlepin.pki.PKIUtility
    for the 1st parameter of org.candlepin.model.ProductCertificateCurator.<init>(ProductCertificateCurator.java:53)
  at org.candlepin.model.ProductCertificateCurator.class(ProductCertificateCurator.java:45)
  while locating org.candlepin.model.ProductCertificateCurator
Caused by: java.io.IOException (same stack trace as error #3)
4 errors
	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:554)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:188)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:56)
	at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5127)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5643)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

candlepin’s error.log

2022-02-01 10:31:59,604 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2022-02-01 10:31:59,659 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2022-02-01 10:31:59,659 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2022-02-01 10:32:00,699 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.metamodel.internal.MetadataContext - HHH015011: Unable to locate static metamodel field : org.candlepin.model.KeyPair_#privateKey; this may or may not indicate a problem with the static metamodel
2022-02-01 10:32:00,699 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.metamodel.internal.MetadataContext - HHH015011: Unable to locate static metamodel field : org.candlepin.model.KeyPair_#publicKey; this may or may not indicate a problem with the static metamodel

health check output:

Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check number of fact names in database:                               e[32me[1m[OK]e[0m
--------------------------------------------------------------------------------
Check whether all services are running:                               e[32me[1m[OK]e[0m
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           e[31me[1m[FAIL]e[0m
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)] 
                                                                                
Restart applicable services: 

Stopping the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy

                                                                                
| 
                                                                                
/ 
                                                                                
/ stopping foreman-proxy
                                                                                
/ stopping dynflow-sidekiq@worker-1
                                                                                
/ stopping dynflow-sidekiq@worker-hosts-queue-1
                                                                                
/ stopping dynflow-sidekiq@orchestrator
                                                                                
/ stopping foreman
                                                                                
/ stopping httpd
                                                                                
/ stopping puppetserver
Warning: Stopping foreman.service, but it can still be activated by:
  foreman.socket

                                                                                
/ stopping pulpcore-worker@1.service
                                                                                
/ stopping pulpcore-worker@2.service
                                                                                
/ stopping pulpcore-worker@3.service
                                                                                
/ stopping pulpcore-worker@4.service
                                                                                
/ stopping pulpcore-worker@5.service
                                                                                
/ stopping pulpcore-worker@6.service
                                                                                
/ stopping pulpcore-worker@7.service
                                                                                
/ stopping pulpcore-worker@8.service
                                                                                
/ stopping tomcat
                                                                                
/ stopping postgresql
                                                                                
/ stopping pulpcore-api
                                                                                
/ stopping pulpcore-content
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Warning: Stopping pulpcore-api.service, but it can still be activated by:
  pulpcore-api.socket

Warning: Stopping pulpcore-content.service, but it can still be activated by:
  pulpcore-content.socket

                                                                                
/ stopping rh-redis5-redis
                                                                                
/ All services stopped

Starting the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy

                                                                                
/ All services stopped
                                                                                
- 
                                                                                
- starting rh-redis5-redis
                                                                                
- starting postgresql
                                                                                
- starting pulpcore-api
                                                                                
- starting pulpcore-content
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.

                                                                                
- starting pulpcore-worker@1.service
                                                                                
- starting pulpcore-worker@2.service
                                                                                
- starting pulpcore-worker@3.service
                                                                                
- starting pulpcore-worker@4.service
                                                                                
- starting pulpcore-worker@5.service
                                                                                
- starting pulpcore-worker@6.service
                                                                                
- starting pulpcore-worker@7.service
                                                                                
- starting pulpcore-worker@8.service
                                                                                
- starting tomcat
                                                                                
- starting dynflow-sidekiq@orchestrator
                                                                                
- starting foreman
                                                                                
- starting httpd
                                                                                
- starting puppetserver
                                                                                
- starting dynflow-sidekiq@worker-1
                                                                                
- starting dynflow-sidekiq@worker-hosts-queue-1
                                                                                
- starting foreman-proxy
                                                                                
- All services started

                                                                                
- All services started
                                                                                
\ Checking server response
                                                                                
\ Try 1/5: checking status of hammer ping
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass

                                                                                
\ Waiting 30 seconds before retry.
                                                                                
\ Try 2/5: checking status of hammer ping
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass

                                                                                
\ Waiting 30 seconds before retry.
                                                                                
\ Try 3/5: checking status of hammer ping
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass

                                                                                
\ Waiting 30 seconds before retry.
                                                                                
\ Try 4/5: checking status of hammer ping
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass

                                                                                
\ Waiting 30 seconds before retry.
                                                                                
\ Try 5/5: checking status of hammer ping                             e[31me[1m[FAIL]e[0m
Server response check failed!
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check whether all services are running using the ping call:           e[31me[1m[FAIL]e[0m
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)] 
                                                                                
Check for paused tasks:                                               e[32me[1m[OK]e[0m
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      e[32me[1m[OK]e[0m
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario::FilteredScenario] failed.

The following steps ended up in failing state:

  [server-ping]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="server-ping"

foreman rake for the given error on the page

2022-02-01T15:09:12 [I|app|71a9921d] Started GET "/" for 10.231.104.125 at 2022-02-01 15:09:12 -0500
2022-02-01T15:09:18 [W|app|71a9921d] 404 Not Found
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
 71a9921d | /opt/rh/rh-ruby27/root/usr/share/ruby/net/http.rb:933:in `start'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/ping.rb:275:in `backend_status'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/ping.rb:95:in `block in ping_candlepin_without_auth'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/ping.rb:144:in `exception_watch'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/ping.rb:94:in `ping_candlepin_without_auth'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/ping.rb:245:in `ping_services_for_capsule'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/ping.rb:25:in `ping!'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/services/katello/organization_creator.rb:14:in `create_all_organizations!'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/lib/katello/middleware/organization_created_enforcer.rb:12:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/lib/katello/middleware/event_daemon.rb:10:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in `run_callbacks'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/lib/katello/prevent_json_parsing.rb:12:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/configuration.rb:249:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/request.rb:77:in `block in handle_request'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/request.rb:76:in `handle_request'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/server.rb:447:in `process_client'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-02-01T15:09:18 [I|app|71a9921d] Processing by DashboardController#index as HTML
2022-02-01T15:09:18 [I|app|71a9921d]   Rendering dashboard/index.html.erb within layouts/application
2022-02-01T15:09:18 [I|app|71a9921d]   Rendered dashboard/index.html.erb within layouts/application (Duration: 63.3ms | Allocations: 3607)
2022-02-01T15:09:18 [W|app|71a9921d] Katello::Errors::CandlepinNotRunning
2022-02-01T15:09:18 [I|app|71a9921d] Backtrace for 'Katello::Errors::CandlepinNotRunning' error (ActionView::Template::Error): Katello::Errors::CandlepinNotRunning
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/katello/resources/candlepin.rb:27:in `raise_rest_client_exception'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/katello/http_resource.rb:94:in `rescue in issue_request'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/katello/http_resource.rb:84:in `issue_request'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/katello/http_resource.rb:47:in `block (2 levels) in singleton class'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/katello/resources/candlepin/owner.rb:32:in `find'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/glue/candlepin/owner.rb:20:in `find_owner'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/glue/candlepin/owner.rb:31:in `owner_details'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/glue/candlepin/owner.rb:46:in `content_access_mode'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/glue/candlepin/owner.rb:51:in `block in simple_content_access?'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache.rb:332:in `block in fetch'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache.rb:717:in `block in save_block_result_to_cache'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache.rb:686:in `block in instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:182:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache.rb:686:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache.rb:716:in `save_block_result_to_cache'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache.rb:332:in `fetch'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/glue/candlepin/owner.rb:50:in `simple_content_access?'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/helpers/katello/concerns/dashboard_helper_extensions.rb:35:in `removed_widgets'
 71a9921d | /usr/share/foreman/app/helpers/dashboard_helper.rb:32:in `widgets_to_add'
 71a9921d | /usr/share/foreman/app/helpers/dashboard_helper.rb:16:in `block in dashboard_actions'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/helpers/capture_helper.rb:45:in `block in capture'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/helpers/capture_helper.rb:209:in `with_output_buffer'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/helpers/capture_helper.rb:45:in `capture'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/helpers/tag_helper.rb:273:in `content_tag'
 71a9921d | /usr/share/foreman/app/helpers/dashboard_helper.rb:15:in `dashboard_actions'
 71a9921d | /usr/share/foreman/app/views/dashboard/index.html.erb:2:in `_cb796603fa96e8c93a8fff55bf61fdf3'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/base.rb:274:in `_run'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:185:in `block in render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:182:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:385:in `instrument_render_template'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:183:in `render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:58:in `block (2 levels) in render_template'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:87:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:57:in `block in render_template'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:65:in `render_with_layout'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:56:in `render_template'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:13:in `render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:117:in `block in _render_template'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/base.rb:304:in `in_rendering_context'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:116:in `_render_template'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/streaming.rb:218:in `_render_template'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:103:in `render_to_body'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/rendering.rb:25:in `render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:36:in `render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 71a9921d | /opt/rh/rh-ruby27/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:43:in `render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in `process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 71a9921d | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 71a9921d | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 71a9921d | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in `process'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in `process'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in `dispatch'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in `dispatch'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `each'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/lib/katello/middleware/event_daemon.rb:10:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in `run_callbacks'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 71a9921d | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/lib/katello/prevent_json_parsing.rb:12:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/configuration.rb:249:in `call'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/request.rb:77:in `block in handle_request'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/request.rb:76:in `handle_request'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/server.rb:447:in `process_client'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.0/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 71a9921d | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-02-01T15:09:18 [I|app|71a9921d]   Rendering common/500.html.erb within layouts/application
2022-02-01T15:09:18 [I|app|71a9921d]   Rendered common/500.html.erb within layouts/application (Duration: 4.3ms | Allocations: 1064)
2022-02-01T15:09:18 [I|app|71a9921d]   Rendered layouts/_application_content.html.erb (Duration: 0.9ms | Allocations: 401)
2022-02-01T15:09:18 [I|app|71a9921d]   Rendering layouts/base.html.erb
2022-02-01T15:09:18 [I|app|71a9921d]   Rendered layouts/base.html.erb (Duration: 4.9ms | Allocations: 2548)
2022-02-01T15:09:18 [I|app|71a9921d] Completed 500 Internal Server Error in 140ms (Views: 13.4ms | ActiveRecord: 8.0ms | Allocations: 37924)

Foreman version: 3.1.1
Plugins: 
 - foreman-tasks 5.2.0
 - foreman_remote_execution 5.0.1
 - katello 4.3.0

I have the same issue after migration, AlmaLinux 8.5 as OS

Additional info:

The error on the page is

Katello::Errors::CandlepinNotRunning

catalina log shows that candlepin is killed shortly after Tomcat starts up

Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.CdnCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.ContentCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:38 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public java.lang.Iterable org.candlepin.resource.OwnerResource.createBatchContent(java.lang.String,java.util.List)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@6ab4e45]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Feb 02, 2022 1:09:43 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Feb 02, 2022 1:09:43 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/candlepin] startup failed due to previous errors
Feb 02, 2022 1:09:43 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
SEVERE: The web application [/candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Feb 02, 2022 1:09:43 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->30x048am1g25ur918ma1k1|2e3073fe]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak.
Feb 02, 2022 1:09:43 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->30x048am1g25ur918ma1k1|2e3073fe]-HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak.
Feb 02, 2022 1:09:43 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->30x048am1g25ur918ma1k1|2e3073fe]-HelperThread-#1] but has failed to stop it. This is very likely to create a memory leak.
Feb 02, 2022 1:09:43 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->30x048am1g25ur918ma1k1|2e3073fe]-HelperThread-#2] but has failed to stop it. This is very likely to create a memory leak.
Feb 02, 2022 1:09:43 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5d4b5208]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(1672747794<open>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Feb 02, 2022 1:09:43 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/candlepin has finished in 10,550 ms
Feb 02, 2022 1:09:43 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-127.0.0.1-23443"]
Feb 02, 2022 1:09:43 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 10622 ms

is this a FIPS enabled system?

Yes, it’s on a FIPS enabled system.

Yepp, those are currently broken. (was also reported at 2029914 – FIPS enabled RHEL7 server: Candlepin services not running after installation)

We’re working on it currently with @ekohl. Once the fixes land in nightly, we can see about backporting them to Foreman 3.1.

Thank you, that makes a lot of sense, I saw bug reports from 2017 but thought it would have been covered in 3.1.

If you don’t mind, can you tell me what is the latest version that is FIPS complaint?

I think the above bug is not present in 3.0.

However, bear in mind, neither CentOS nor Foreman are FIPS certified, so I don’t see much value in running this in FIPS mode at all :wink:

FTR, this is fixed in nightly, so 3.2/4.4 should run on EL7/FIPS again.