Requirements for foreman katello

So is it running now?

Hi @gvde, Still that candlepin service is not getting started…

With the same exception in the catalina log? I would suspect it’s a different error now…

As before: stop tomcat.service, make sure it’s stopped, start it again and post the latest logs.

Please find the logs:

cat /var/log/tomcat/catalina.2025-01-06.log

06-Jan-2025 10:00:03.456 INFO [main] liquibase.database.null Set default schema name to public
06-Jan-2025 10:00:03.474 INFO [main] liquibase.changelog.null Reading from public.databasechangelog
06-Jan-2025 10:00:04.170 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.186 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.188 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.OwnerCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.200 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.200 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.200 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.209 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.210 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.230 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.258 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:04.258 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:00:06.932 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
06-Jan-2025 10:00:06.935 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
06-Jan-2025 10:00:06.938 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jan-2025 10:00:06.939 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gdwnvt5toz97|4a0b3f5b]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 java.base@17.0.13/java.util.TimerThread.mainLoop(Timer.java:563)
 java.base@17.0.13/java.util.TimerThread.run(Timer.java:516)
06-Jan-2025 10:00:06.939 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gdwnvt5toz97|4a0b3f5b]-HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:00:06.939 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gdwnvt5toz97|4a0b3f5b]-HelperThread-#1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:00:06.940 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gdwnvt5toz97|4a0b3f5b]-HelperThread-#2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:00:06.940 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7c6923fe]) and a value of type [liquibase.SingletonScopeManager] (value [liquibase.SingletonScopeManager@4e9fa88e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 10:00:06.941 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@72d11900]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(2138130522<open>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 10:00:06.945 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/candlepin] has finished in [7,977] ms
06-Jan-2025 10:00:06.947 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [8007] milliseconds
06-Jan-2025 10:04:27.912 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:04:27.913 INFO [Thread-5] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
06-Jan-2025 10:04:27.914 INFO [Thread-5] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:04:27.915 INFO [Thread-5] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:04:28.216 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
06-Jan-2025 10:04:28.389 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:04:28.494 WARNING [main] org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the [ciphers] attribute in a manner consistent with the latest OpenSSL development branch. Some of the specified [ciphers] are not supported by the configured SSL engine for this connector (which may use JSSE or an older OpenSSL version) and have been skipped: [[TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256]]
06-Jan-2025 10:04:28.495 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["https-jsse-nio-127.0.0.1-23443"]]
        org.apache.catalina.LifecycleException: Protocol handler initialization failed
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1011)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardService.initInternal(StandardService.java:554)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1046)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:686)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:709)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:569)
                at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
        Caused by: java.lang.IllegalArgumentException: /etc/candlepin/certs/keystore (Permission denied)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:115)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:228)
                at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1334)
                at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1347)
                at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:654)
                at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:75)
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1009)
                ... 13 more
        Caused by: java.io.FileNotFoundException: /etc/candlepin/certs/keystore (Permission denied)
                at java.base/java.io.FileInputStream.open0(Native Method)
                at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
                at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
                at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
                at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
                at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
                at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:118)
                at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:210)
                at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:254)
                at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:308)
                at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:268)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:113)
                ... 20 more
06-Jan-2025 10:04:28.496 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [406] milliseconds
06-Jan-2025 10:04:28.518 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
06-Jan-2025 10:04:28.518 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
06-Jan-2025 10:04:28.522 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
06-Jan-2025 10:04:31.375 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
06-Jan-2025 10:04:33.009 INFO [main] liquibase.database.null Set default schema name to public
06-Jan-2025 10:04:33.028 INFO [main] liquibase.changelog.null Reading from public.databasechangelog
06-Jan-2025 10:04:33.721 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.736 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.739 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.OwnerCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.750 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.750 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.751 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.760 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.760 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.782 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.811 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:33.811 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@bf2c325]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:04:36.425 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
06-Jan-2025 10:04:36.428 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
06-Jan-2025 10:04:36.431 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jan-2025 10:04:36.431 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8ge2ful1gmpt5k|36d9c340]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 java.base@17.0.13/java.util.TimerThread.mainLoop(Timer.java:563)
 java.base@17.0.13/java.util.TimerThread.run(Timer.java:516)
06-Jan-2025 10:04:36.432 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8ge2ful1gmpt5k|36d9c340]-HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:04:36.432 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8ge2ful1gmpt5k|36d9c340]-HelperThread-#1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:04:36.432 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8ge2ful1gmpt5k|36d9c340]-HelperThread-#2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:04:36.433 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@21b0b73]) and a value of type [liquibase.SingletonScopeManager] (value [liquibase.SingletonScopeManager@879a237]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 10:04:36.433 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@552fe596]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(1078244431<open>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 10:04:36.437 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/candlepin] has finished in [7,915] ms
06-Jan-2025 10:04:36.439 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [7943] milliseconds
06-Jan-2025 10:09:34.944 INFO [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8ge2ful1gmpt5k|36d9c340]-AdminTaskTimer] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1349)
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1337)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1174)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1141)
                at com.mchange.v2.resourcepool.BasicResourcePool.checkIdleResources(BasicResourcePool.java:1673)
                at com.mchange.v2.resourcepool.BasicResourcePool.access$2000(BasicResourcePool.java:44)
                at com.mchange.v2.resourcepool.BasicResourcePool$CheckIdleResourcesTask.run(BasicResourcePool.java:2214)
                at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
                at java.base/java.util.TimerThread.run(Timer.java:516)
06-Jan-2025 10:11:11.829 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:11:11.830 INFO [Thread-5] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
06-Jan-2025 10:11:11.831 INFO [Thread-5] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:11:11.831 INFO [Thread-5] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:11:47.581 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
06-Jan-2025 10:11:47.757 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 10:11:47.866 WARNING [main] org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the [ciphers] attribute in a manner consistent with the latest OpenSSL development branch. Some of the specified [ciphers] are not supported by the configured SSL engine for this connector (which may use JSSE or an older OpenSSL version) and have been skipped: [[TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256]]
06-Jan-2025 10:11:47.867 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["https-jsse-nio-127.0.0.1-23443"]]
        org.apache.catalina.LifecycleException: Protocol handler initialization failed
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1011)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardService.initInternal(StandardService.java:554)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1046)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:686)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:709)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:569)
                at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
        Caused by: java.lang.IllegalArgumentException: /etc/candlepin/certs/keystore (Permission denied)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:115)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:228)
                at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1334)
                at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1347)
                at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:654)
                at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:75)
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1009)
                ... 13 more
        Caused by: java.io.FileNotFoundException: /etc/candlepin/certs/keystore (Permission denied)
                at java.base/java.io.FileInputStream.open0(Native Method)
                at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
                at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
                at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
                at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
                at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
                at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:118)
                at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:210)
                at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:254)
                at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:308)
                at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:268)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:113)
                ... 20 more
06-Jan-2025 10:11:47.867 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [407] milliseconds
06-Jan-2025 10:11:47.890 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
06-Jan-2025 10:11:47.890 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
06-Jan-2025 10:11:47.893 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
06-Jan-2025 10:11:50.709 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
06-Jan-2025 10:11:52.313 INFO [main] liquibase.database.null Set default schema name to public
06-Jan-2025 10:11:52.331 INFO [main] liquibase.changelog.null Reading from public.databasechangelog
06-Jan-2025 10:11:53.026 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.042 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.044 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.OwnerCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.056 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.056 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.056 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.066 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.066 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.086 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.114 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:53.114 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2314f7a7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 10:11:55.814 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
06-Jan-2025 10:11:55.816 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
06-Jan-2025 10:11:55.819 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jan-2025 10:11:55.820 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gebutr1v1a67e|bcef6b4]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 java.base@17.0.13/java.util.TimerThread.mainLoop(Timer.java:563)
 java.base@17.0.13/java.util.TimerThread.run(Timer.java:516)
06-Jan-2025 10:11:55.820 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gebutr1v1a67e|bcef6b4]-HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:11:55.820 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gebutr1v1a67e|bcef6b4]-HelperThread-#1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:11:55.821 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8gebutr1v1a67e|bcef6b4]-HelperThread-#2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 10:11:55.821 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@72d11900]) and a value of type [liquibase.SingletonScopeManager] (value [liquibase.SingletonScopeManager@7f71485a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 10:11:55.821 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4a32e299]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(2036000905<open>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 10:11:55.827 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/candlepin] has finished in [7,933] ms
06-Jan-2025 10:11:55.829 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [7961] milliseconds

cat /var/log/candlepin/candlepin.log
2025-01-06 10:00:01,982 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-06 10:00:01,983 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-06 10:00:01,988 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-06 10:00:02,033 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-06 10:00:02,041 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-06 10:00:02,046 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-06 10:00:02,115 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-06 10:00:03,513 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-06 10:00:03,667 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-06 10:00:04,019 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-06 10:00:04,019 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-06 10:00:04,020 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-06 10:00:04,020 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-06 10:00:04,020 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-06 10:00:04,020 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-06 10:00:04,021 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-06 10:00:04,021 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-06 10:00:04,021 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-06 10:00:04,022 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-06 10:00:04,022 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-06 10:00:04,022 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-06 10:00:04,022 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-06 10:00:04,023 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-06 10:00:04,023 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-06 10:00:04,023 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-06 10:00:04,023 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-06 10:00:04,024 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-06 10:00:04,024 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-06 10:00:04,024 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-06 10:00:04,025 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-06 10:00:04,025 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-06 10:00:04,025 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-06 10:00:05,707 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-06 10:00:05,824 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-06 10:00:05,824 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-06 10:00:06,671 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381
2025-01-06 10:04:31,541 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-06 10:04:31,543 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-06 10:04:31,547 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-06 10:04:31,590 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-06 10:04:31,599 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-06 10:04:31,604 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-06 10:04:31,672 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-06 10:04:33,066 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-06 10:04:33,218 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-06 10:04:33,569 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-06 10:04:33,570 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-06 10:04:33,570 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-06 10:04:33,570 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-06 10:04:33,570 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-06 10:04:33,571 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-06 10:04:33,571 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-06 10:04:33,571 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-06 10:04:33,572 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-06 10:04:33,572 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-06 10:04:33,572 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-06 10:04:33,572 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-06 10:04:33,573 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-06 10:04:33,573 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-06 10:04:33,573 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-06 10:04:33,574 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-06 10:04:33,574 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-06 10:04:33,574 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-06 10:04:33,575 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-06 10:04:33,575 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-06 10:04:33,575 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-06 10:04:33,575 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-06 10:04:33,576 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-06 10:04:35,223 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-06 10:04:35,338 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-06 10:04:35,338 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-06 10:04:36,163 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381
2025-01-06 10:11:50,871 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-06 10:11:50,872 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-06 10:11:50,877 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-06 10:11:50,918 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-06 10:11:50,926 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-06 10:11:50,931 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-06 10:11:50,997 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-06 10:11:52,372 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-06 10:11:52,524 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-06 10:11:52,874 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-06 10:11:52,875 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-06 10:11:52,875 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-06 10:11:52,875 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-06 10:11:52,876 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-06 10:11:52,876 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-06 10:11:52,876 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-06 10:11:52,876 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-06 10:11:52,877 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-06 10:11:52,877 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-06 10:11:52,877 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-06 10:11:52,878 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-06 10:11:52,878 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-06 10:11:52,878 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-06 10:11:52,878 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-06 10:11:52,879 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-06 10:11:52,879 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-06 10:11:52,879 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-06 10:11:52,880 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-06 10:11:52,880 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-06 10:11:52,880 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-06 10:11:52,880 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-06 10:11:52,881 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-06 10:11:54,557 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-06 10:11:54,677 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-06 10:11:54,677 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-06 10:11:55,545 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381

It’s again the same error. What are the permissions in /etc/candlepin/certs/ now? Did they change again?

Which user/group is the tomcat process running?

# systemctl status tomcat.service
● tomcat.service - Apache Tomcat Web Application Container
     Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; preset: disabled)
     Active: active (running) since Mon 2024-12-30 09:40:28 CET; 1 week 0 days ago
   Main PID: 1194 (java)
      Tasks: 94 (limit: 306353)
...
# ps -o pid,euid,egid,cmd -p 1194
    PID  EUID  EGID CMD
   1194    91    91 /usr/lib/jvm/jre-17/bin/java -Xms1024m -Xmx4096m -Dcom.redhat.fips=false -Djava
# id tomcat
uid=91(tomcat) gid=91(tomcat) groups=91(tomcat)
# rpm -V tomcat
SM5....T.  c /etc/tomcat/server.xml
S.5....T.  c /etc/tomcat/tomcat.conf
# systemctl cat tomcat.service
# /usr/lib/systemd/system/tomcat.service
# Systemd unit file for default tomcat
# 
# To create clones of this service:
# DO NOTHING, use tomcat@.service instead.

[Unit]
Description=Apache Tomcat Web Application Container
After=syslog.target network.target

[Service]
Type=simple
EnvironmentFile=/etc/tomcat/tomcat.conf
Environment="NAME="
EnvironmentFile=-/etc/sysconfig/tomcat
ExecStart=/usr/libexec/tomcat/server start
SuccessExitStatus=143
User=tomcat

[Install]
WantedBy=multi-user.target

Please find the requested details as below:

#ls -la /etc/candlepin/certs/
total 60
drwxr-xr--. 3 root root    4096 Jan  3 12:27 .
drwxr-xr-x. 3 root root      89 Jan  6 09:48 ..
-r--r-----. 1 root tomcat  2508 Jan  3 12:27 candlepin-ca.crt
-r--r-----. 1 root tomcat 11148 Jan  3 12:27 candlepin-ca.key
-rw-r-----. 1 root tomcat  4808 Jan  3 12:27 keystore
-r--r-----. 1 root tomcat    32 Jan  3 12:20 keystore_password-file
-rw-r-----. 1 root tomcat  4166 Jan  3 12:27 truststore
-r--r-----. 1 root tomcat    32 Jan  3 12:20 truststore_password-file
drw-r--r--. 2 root root      37 Jan  3 12:20 upstream

# ls -laZ /etc/candlepin/certs/
total 60
drwxr-xr--. 3 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0         4096 Jan  3 12:27 .
drwxr-xr-x. 3 root root   system_u:object_r:candlepin_etc_rw_t:s0                 89 Jan  6 09:48 ..
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0  2508 Jan  3 12:27 candlepin-ca.crt
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0 11148 Jan  3 12:27 candlepin-ca.key
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4808 Jan  3 12:27 keystore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Jan  3 12:20 keystore_password-file
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4166 Jan  3 12:27 truststore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Jan  3 12:20 truststore_password-file
drw-r--r--. 2 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0           37 Jan  3 12:20 upstream




#systemctl status tomcat
● tomcat.service - Apache Tomcat Web Application Container
     Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; preset: disabled)
     Active: active (running) since Mon 2025-01-06 10:25:44 UTC; 57min ago
   Main PID: 4141220 (java)
      Tasks: 35 (limit: 407992)

#ps -o pid,euid,egid,cmd -p 4141220
    PID  EUID  EGID CMD
4141220    53    53 /usr/lib/jvm/jre-17/bin/java -Xms1024m -Xmx4096m -Dcom.redhat.fips=false -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.c

# id tomcat
uid=53(tomcat) gid=53(tomcat) groups=10(wheel),53(tomcat)

# rpm -V tomcat
SM5....T.  c /etc/tomcat/server.xml
S.5....T.  c /etc/tomcat/tomcat.conf


# systemctl cat tomcat.service
# /usr/lib/systemd/system/tomcat.service
# Systemd unit file for default tomcat
#
# To create clones of this service:
# DO NOTHING, use tomcat@.service instead.

[Unit]
Description=Apache Tomcat Web Application Container
After=syslog.target network.target

[Service]
Type=simple
EnvironmentFile=/etc/tomcat/tomcat.conf
Environment="NAME="
EnvironmentFile=-/etc/sysconfig/tomcat
ExecStart=/usr/libexec/tomcat/server start
SuccessExitStatus=143
User=tomcat

[Install]
WantedBy=multi-user.target



Your installation seems to be really messed up:

uid=53(tomcat) gid=53(tomcat) groups=10(wheel),53(tomcat)

The tomcat user is in group wheel. How did it get in there? That’s not something to expect if you have installed foreman on a new server.

As you seemed to have run multiple foreman-installer attempts before and you have removed ssl-build in between etc. I can only highly recommend to reset once, reinstall the os on the server and start again following the docs.

There seem to be so many local, custom changes to your system which possibly mess with the system in an unexpected way.

Same here:

Easily to miss: there is the o=x mode bit missing. I guess, foreman-installer didn’t expect someone to change the directory permissions thus doesn’t check and enforce it. Thus, user/group tomcat cannot read it.

Why does the directory have different permissions? And how did it happen, that you had the keystore file with different permissions before? foreman-installer sets it correctly thus someone/something must have changed the permissions later. And considering those strange, unexpected changes to the system it’s really hard to guess what else may have been broken for whatever reason. I don’t see why foreman-installer would do that. I suspect someone/something else did it.

This is how the content of /etc/candlepin should look like. You can run ls as normal user (not root) and should see everything. If permissions, owner/group and contexts fit it should be able to read the necessary files.

$ ls -laRZ /etc/candlepin/
/etc/candlepin/:
total 24
drwxr-xr-x.   3 root   root   system_u:object_r:candlepin_etc_rw_t:s0         59 Nov 19 19:02 .
drwxr-xr-x. 134 root   root   system_u:object_r:etc_t:s0                    8192 Dec 30 09:40 ..
-rw-r-----.   1 tomcat tomcat system_u:object_r:candlepin_etc_rw_t:s0       7204 Jun 21  2024 broker.xml
-rw-r-----.   1 root   tomcat system_u:object_r:candlepin_etc_rw_t:s0       1674 Nov 19 19:02 candlepin.conf
drwxr-xr-x.   3 root   root   system_u:object_r:candlepin_etc_certs_rw_t:s0  166 Nov 19 19:02 certs

/etc/candlepin/certs:
total 36
drwxr-xr-x. 3 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0          166 Nov 19 19:02 .
drwxr-xr-x. 3 root root   system_u:object_r:candlepin_etc_rw_t:s0                 59 Nov 19 19:02 ..
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0  2524 Jul 13  2022 candlepin-ca.crt
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0 11151 Oct 20 16:55 candlepin-ca.key
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4663 Jan 16  2024 keystore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Nov 19 19:02 keystore_password-file
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4050 Jan 16  2024 truststore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Nov 19 19:02 truststore_password-file
drwxr-xr-x. 2 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0           37 Oct 13  2023 upstream

/etc/candlepin/certs/upstream:
total 4
drwxr-xr-x. 2 root root system_u:object_r:candlepin_etc_certs_rw_t:s0   37 Oct 13  2023 .
drwxr-xr-x. 3 root root system_u:object_r:candlepin_etc_certs_rw_t:s0  166 Nov 19 19:02 ..
-rw-r--r--. 1 root root system_u:object_r:candlepin_etc_certs_rw_t:s0 2472 Nov 25 13:46 candlepin-redhat-ca.crt

But again: most important is to find out how those permissions were broken at all. And how the tomcat user got into group wheel. And what else this “how” may have modified… Because otherwise it won’t be any fun if changes to your system happen which are unexpected and break your foreman installation…

2 Likes

Hi, I recently deployed Foreman 3.13 and Katello 4.15 and had the same SSL issue reported by Priya1.
I had proper CA signed SSL certs generated and ready to go for the installation, but when I used them the install failed.,This put the install in a very broken state (every foreman-installer command failed with SSL errors), so I rolled back the machine.

Second time around I installed with default certs, then afterwards once Foreman was working I updated the install with the proper signed certs, via :foreman-installer --scenario katello --certs-server-cert "{{ foreman_ssl_certificate }}" --certs-server-key "{{ foreman_ssl_private_key }}" --certs-server-ca-cert "{{ foreman_ssl_ca_certificate }}" --certs-update-server --certs-update-server-ca

This worked ok, but the next day when I went to add extra components to foreman it also failed with the same SSL issue.

I solved the problem after finding this thread, backing up and deleting the contents of /root/ssl-build and then running
foreman-installer --scenario-katello --certs-update-all

The SSL issue sounds like a bug.
I unfortunately don’t have anything to contribute RE the candlepin issue that Priya was seeing.

Please open a new thread. This has most likely nothing to do with this thread. It doesn’t help to add seemingly similar topics here. Open a new thread and include all the necessary information.