Katello 4.0 installation on RHEL 8 failed

Problem:
Fresh installation of katello 4.0 on a rhel8 system failed with foreman-rake db:seed’ returned 1 instead of one of [0].

Expected outcome:
Successfull installation of katello.

Foreman and Proxy versions:
Foreman 2.4 and Katello 4.0

Foreman and Proxy plugin versions:

Distribution and version:
Red Hat Enterprise Linux 8.3 (Ootpa)

Other relevant data:
I followed th instruction from https://community.theforeman.org/t/what-is-the-state-of-katello-support-on-el8/22186/6

[root@scotty ~]# foreman-installer -v --scenario katello 
2021-05-03 08:37:24 [NOTICE] [root] Loading default values from puppet modules...
2021-05-03 08:37:36 [NOTICE] [root] ... finished
2021-05-03 08:37:47 [NOTICE] [root] Running validation checks
2021-05-03 08:38:17 [NOTICE] [configure] Starting system configuration.
The total number of configuration tasks may increase during the run.
Observe logs or specify --verbose-log-level to see individual configuration tasks.
2021-05-03 08:38:53 [NOTICE] [configure] 100 out of 1944 done.
2021-05-03 08:38:53 [NOTICE] [configure] 200 out of 1944 done.
2021-05-03 08:39:08 [NOTICE] [configure] 300 out of 1944 done.
2021-05-03 08:39:23 [NOTICE] [configure] 400 out of 1944 done.
2021-05-03 08:39:27 [NOTICE] [configure] 500 out of 1944 done.
2021-05-03 08:39:28 [NOTICE] [configure] 600 out of 1946 done.
2021-05-03 08:39:28 [NOTICE] [configure] 700 out of 1946 done.
2021-05-03 08:39:29 [NOTICE] [configure] 800 out of 1948 done.
2021-05-03 08:39:32 [NOTICE] [configure] 900 out of 1949 done.
2021-05-03 08:39:33 [NOTICE] [configure] 1000 out of 1951 done.
2021-05-03 08:39:33 [NOTICE] [configure] 1100 out of 1954 done.
2021-05-03 08:39:35 [NOTICE] [configure] 1200 out of 1955 done.
2021-05-03 08:39:39 [NOTICE] [configure] 1300 out of 1956 done.
2021-05-03 08:42:23 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-05-03 08:42:23 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-05-03 08:42:23 [NOTICE] [configure] 1400 out of 1956 done.
2021-05-03 08:42:24 [NOTICE] [configure] 1500 out of 1956 done.
2021-05-03 08:42:24 [NOTICE] [configure] 1600 out of 1956 done.
2021-05-03 08:43:17 [NOTICE] [configure] 1700 out of 1956 done.
2021-05-03 08:43:18 [NOTICE] [configure] 1800 out of 1956 done.
2021-05-03 08:43:42 [NOTICE] [configure] 1900 out of 1956 done.
2021-05-03 08:43:55 [NOTICE] [configure] System configuration has finished.

There were errors detected during install.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/katello.log
[root@scotty ~]#

From katello.log:

2021-05-03 08:39:41 [DEBUG ] [configure] Foreman::Rake[db:migrate]: Starting to evaluate the resource (1384 of 1956)
2021-05-03 08:39:41 [DEBUG ] [configure] Foreman::Rake[db:migrate]: Evaluated in 0.00 seconds
2021-05-03 08:39:41 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Starting to evaluate the resource (1385 of 1956)
2021-05-03 08:39:41 [DEBUG ] [configure] Exec[foreman-rake-db:migrate](provider=posix): Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
2021-05-03 08:39:41 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
2021-05-03 08:40:36 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' won't be executed because of failed check 'unless'
2021-05-03 08:40:36 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Evaluated in 55.11 seconds
2021-05-03 08:40:36 [DEBUG ] [configure] Foreman::Rake[db:migrate]: Starting to evaluate the resource (1386 of 1956)
2021-05-03 08:40:36 [DEBUG ] [configure] Foreman::Rake[db:migrate]: Evaluated in 0.00 seconds
2021-05-03 08:40:36 [DEBUG ] [configure] Prefetching cli resources for foreman_config_entry
2021-05-03 08:40:36 [DEBUG ] [configure] Executing with uid=foreman gid=foreman: '/usr/sbin/foreman-rake -- config '
2021-05-03 08:41:24 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Starting to evaluate the resource (1387 of 1956)
2021-05-03 08:41:24 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]/value: value changed 'true' to 'false'
2021-05-03 08:41:24 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: The container Class[Foreman::Database] will propagate my refresh event
2021-05-03 08:41:24 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Scheduling refresh of Foreman::Rake[db:seed]
2021-05-03 08:41:24 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Evaluated in 0.06 seconds
2021-05-03 08:41:24 [DEBUG ] [configure] Foreman::Rake[db:seed]: Starting to evaluate the resource (1388 of 1956)
2021-05-03 08:41:24 [DEBUG ] [configure] Foreman::Rake[db:seed]: Scheduling refresh of Exec[foreman-rake-db:seed]
2021-05-03 08:41:24 [DEBUG ] [configure] Foreman::Rake[db:seed]: Evaluated in 0.00 seconds
2021-05-03 08:41:24 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Starting to evaluate the resource (1389 of 1956)
2021-05-03 08:41:24 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' won't be executed because of failed check 'refreshonly'
2021-05-03 08:41:24 [DEBUG ] [configure] Exec[foreman-rake-db:seed](provider=posix): Executing '/usr/sbin/foreman-rake db:seed'
2021-05-03 08:41:24 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:seed'
2021-05-03 08:42:23 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted!
2021-05-03 08:42:23 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: There was an issue with the backend service candlepin: 404 Not Found
2021-05-03 08:42:23 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/backend_services_check.rb:1
7:in `block in plan'
.....

Got the same problem here. Doing a fresh installation on Almalinux 8.3 (as a test to replace CentOS) using external PostgreSQL 12.6 databases for candlepin, pulpcore and foreman.
Foreman running on a dedicated Almalinux server (VM).
Product versions Katello 4.0 and Foreman 2.4. Following the instructions in The Foreman Documentation | Foreman 2.4 and Katello 4.0 documentation
Please find below my installation report.

2021-05-04 09:34:14 [NOTICE] [root] Loading default values from puppet modules...
2021-05-04 09:34:23 [NOTICE] [root] ... finished
2021-05-04 09:34:28 [NOTICE] [root] Running validation checks
2021-05-04 09:34:46 [NOTICE] [configure] Starting system configuration.
  The total number of configuration tasks may increase during the run.
  Observe logs or specify --verbose-log-level to see individual configuration tasks.
2021-05-04 09:35:12 [NOTICE] [configure] 100 out of 1769 done.
2021-05-04 09:35:13 [NOTICE] [configure] 200 out of 1769 done.
2021-05-04 09:35:21 [NOTICE] [configure] 300 out of 1769 done.
2021-05-04 09:35:22 [NOTICE] [configure] 400 out of 1769 done.
2021-05-04 09:35:25 [NOTICE] [configure] 500 out of 1770 done.
2021-05-04 09:35:25 [NOTICE] [configure] 600 out of 1771 done.
2021-05-04 09:35:26 [NOTICE] [configure] 700 out of 1771 done.
2021-05-04 09:35:31 [NOTICE] [configure] 800 out of 1772 done.
2021-05-04 09:35:32 [NOTICE] [configure] 900 out of 1773 done.
2021-05-04 09:35:32 [NOTICE] [configure] 1000 out of 1776 done.
2021-05-04 09:35:35 [NOTICE] [configure] 1100 out of 1778 done.
2021-05-04 09:35:37 [NOTICE] [configure] 1200 out of 1779 done.
2021-05-04 09:38:24 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-05-04 09:38:24 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-05-04 09:38:24 [NOTICE] [configure] 1300 out of 1779 done.
2021-05-04 09:38:25 [NOTICE] [configure] 1400 out of 1779 done.
2021-05-04 09:38:25 [NOTICE] [configure] 1500 out of 1779 done.
2021-05-04 09:38:25 [NOTICE] [configure] 1600 out of 1779 done.
2021-05-04 09:39:32 [NOTICE] [configure] 1700 out of 1779 done.
2021-05-04 09:39:45 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

I have been looking everywhere for a solution, but unfortunately not found and I was wondering whethter Support did provide you with a solution.

rgds,
-gw

Please code blocks when pasting log output. Typically, fenced code blocks are easier and supported.

The critical part is this:

2021-05-03 08:42:23 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted!
2021-05-03 08:42:23 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: There was an issue with the backend service candlepin: 404 Not Found
2021-05-03 08:42:23 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/backend_services_check.rb:1

For some reason Candlepin wasn’t working well. You should really look into that. Note that you can also run the command manually (/usr/sbin/foreman-rake db:seed) to see the exact output, but I think Candlepin logs are in /var/log/candlepin and should tell you more. The output of journalctl -u tomcat can also be informative.

I already run the db:seed command manually, including the trace option, but the output was the same as above.
In /var/log/candlepin I have two files:
1.) error.log

[root@scotty candlepin]# cat error.log
2021-05-04 10:57:16,125 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2021-05-04 10:57:16,321 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2021-05-04 10:57:16,322 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2021-05-04 10:57:21,400 [thread=main] [=, org=, csid=] WARN  org.hibernate.metamodel.internal.MetadataContext - HHH015011: Unable to locate static metamodel field : org.candlepin.model.KeyPair_#publicKey; this may or maynot indicate a problem with the static metamodel
2021-05-04 10:57:21,402 [thread=main] [=, org=, csid=] WARN  org.hibernate.metamodel.internal.MetadataContext - HHH015011: Unable to locate static metamodel field : org.candlepin.model.KeyPair_#privateKey; this may or may not indicate a problem with the static metamodel
2021-05-04 10:57:27,025 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.LoggingListener in AddressSettings
2021-05-04 10:57:27,026 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.LoggingListener in AddressSettings
2021-05-04 10:57:27,108 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.ActivationListener in AddressSettings
2021-05-04 10:57:27,109 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.ActivationListener in AddressSettings
2021-05-04 10:57:27,135 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue katello_candlepin_event_monitor.candlepin_events in AddressSettings
2021-05-04 10:57:27,137 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue katello_candlepin_event_monitor.candlepin_events in AddressSettings
2021-05-04 10:57:27,154 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue jobs in AddressSettings
2021-05-04 10:57:27,154 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue jobs in AddressSettings
[root@scotty candlepin]#

2.) candlepin.log

[root@scotty candlepin]# cat candlepin.log
2021-05-04 10:56:57,117 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2021-05-04 10:56:57,151 [thread=main] [=, org=, csid=] INFO  org.candlepin.pki.impl.JSSProviderLoader - Using JSS version 4.7.3
2021-05-04 10:56:57,402 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2021-05-04 10:56:57,413 [thread=main] [=, org=, csid=] INFO  org.candlepin.common.config.EncryptedConfiguration - No secret file provided.
2021-05-04 10:56:57,469 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Running under postgresql
2021-05-04 10:56:57,504 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, insights_auto_register, storage_band, cores, hypervisors_async, org_level_content_access, guest_limit, ram, batch_bind]
2021-05-04 10:56:58,322 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2021-05-04 10:56:59,231 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2021-05-04 10:56:59,233 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CRLUpdateJob: org.candlepin.async.tasks.CRLUpdateJob
2021-05-04 10:56:59,234 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2021-05-04 10:56:59,236 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2021-05-04 10:56:59,237 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2021-05-04 10:56:59,239 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2021-05-04 10:56:59,240 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2021-05-04 10:56:59,242 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2021-05-04 10:56:59,243 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2021-05-04 10:56:59,245 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2021-05-04 10:56:59,246 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2021-05-04 10:56:59,248 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2021-05-04 10:56:59,249 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2021-05-04 10:56:59,251 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: OrphanCleanupJob: org.candlepin.async.tasks.OrphanCleanupJob
2021-05-04 10:56:59,252 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2021-05-04 10:56:59,254 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2021-05-04 10:56:59,255 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2021-05-04 10:56:59,257 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2021-05-04 10:56:59,259 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2021-05-04 10:56:59,260 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2021-05-04 10:57:16,125 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2021-05-04 10:57:16,321 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2021-05-04 10:57:16,322 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2021-05-04 10:57:21,400 [thread=main] [=, org=, csid=] WARN  org.hibernate.metamodel.internal.MetadataContext - HHH015011: Unable to locate static metamodel field : org.candlepin.model.KeyPair_#publicKey; this may or maynot indicate a problem with the static metamodel
2021-05-04 10:57:21,402 [thread=main] [=, org=, csid=] WARN  org.hibernate.metamodel.internal.MetadataContext - HHH015011: Unable to locate static metamodel field : org.candlepin.model.KeyPair_#privateKey; this may or may not indicate a problem with the static metamodel
2021-05-04 10:57:22,466 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2021-05-04 10:57:22.35
2021-05-04 10:57:24,913 [thread=main] [=, org=, csid=] INFO  org.candlepin.messaging.impl.artemis.ArtemisContextListener - Initializing embedded Artemis server...
2021-05-04 10:57:24,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.messaging.impl.artemis.ArtemisContextListener - Loading Artemis config file: /etc/candlepin/broker.xml
2021-05-04 10:57:27,025 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.LoggingListener in AddressSettings
2021-05-04 10:57:27,026 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.LoggingListener in AddressSettings
2021-05-04 10:57:27,108 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.ActivationListener in AddressSettings
2021-05-04 10:57:27,109 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.ActivationListener in AddressSettings
2021-05-04 10:57:27,135 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue katello_candlepin_event_monitor.candlepin_events in AddressSettings
2021-05-04 10:57:27,137 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue katello_candlepin_event_monitor.candlepin_events in AddressSettings
2021-05-04 10:57:27,154 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue jobs in AddressSettings
2021-05-04 10:57:27,154 [thread=main] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue jobs in AddressSettings
2021-05-04 10:57:27,805 [thread=main] [=, org=, csid=] INFO  org.candlepin.messaging.impl.artemis.ArtemisContextListener - Embedded Artemis server started successfully
2021-05-04 10:57:27,806 [thread=main] [=, org=, csid=] INFO  org.candlepin.messaging.impl.artemis.ArtemisSessionFactory - Connecting to Artemis server at vm://0
2021-05-04 10:57:27,874 [thread=main] [=, org=, csid=] INFO  org.candlepin.messaging.impl.artemis.ArtemisSessionFactory - Artemis session factory initialized
2021-05-04 10:57:28,130 [thread=main] [=, org=, csid=] INFO  org.candlepin.controller.ActiveMQStatusMonitor - Connection to ActiveMQ is available.
2021-05-04 10:57:28,133 [thread=main] [=, org=, csid=] INFO  org.candlepin.audit.ArtemisMessageSource - ActiveMQ status has been updated: UNKNOWN:CONNECTED
2021-05-04 10:57:28,134 [thread=main] [=, org=, csid=] INFO  org.candlepin.audit.ArtemisMessageSource - Connecting to message broker and initializing all message listeners.
2021-05-04 10:57:28,416 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Initializing job manager
2021-05-04 10:57:28,911 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "OrphanCleanupJob" with cron schedule: 0 0 3 ? * 1
2021-05-04 10:57:28,927 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "ActiveEntitlementJob" with cron schedule: 0 0 0/1 * * ?
2021-05-04 10:57:28,944 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "CRLUpdateJob" with cron schedule: 0 0 12 * * ?
2021-05-04 10:57:28,972 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "ExpiredPoolsCleanupJob" with cron schedule: 0 0 0/1 * * ?
2021-05-04 10:57:29,004 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "ImportRecordCleanerJob" with cron schedule: 0 0 12 * * ?
2021-05-04 10:57:29,022 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "JobCleaner" with cron schedule: 0 0 12 * * ?
2021-05-04 10:57:29,039 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "ManifestCleanerJob" with cron schedule: 0 0 12 * * ?
2021-05-04 10:57:29,055 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Scheduled job "UnmappedGuestEntitlementCleanerJob" with cron schedule: 0 0 3/12 * * ?
2021-05-04 10:57:29,060 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobMessageReceiver - Creating 10 job receiver threads with filter: null
2021-05-04 10:57:29,369 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Job manager initialization complete
2021-05-04 10:57:29,369 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Job manager started
[root@scotty candlepin]#

So, there are “only” warnings and I cannot see any blocker.

The tomcat.log seems to be suspicious, I think:

May 04 10:56:40 scotty.home.petersen20.de systemd[1]: Started Apache Tomcat Web Application Container.
May 04 10:56:40 scotty.home.petersen20.de server[638582]: Java virtual machine used: /usr/lib/jvm/jre-11/bin/java
May 04 10:56:40 scotty.home.petersen20.de server[638582]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar
May 04 10:56:40 scotty.home.petersen20.de server[638582]: main class used: org.apache.catalina.startup.Bootstrap
May 04 10:56:40 scotty.home.petersen20.de server[638582]: flags used: -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config
May 04 10:56:40 scotty.home.petersen20.de server[638582]: options used: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
May 04 10:56:40 scotty.home.petersen20.de server[638582]: arguments used: start
May 04 10:56:41 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:41.767 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslProtocols' to 'TLSv1.2' did not find a matching property.
May 04 10:56:41 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:41.980 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
May 04 10:56:41 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:41.981 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
May 04 10:56:41 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:41.995 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
May 04 10:56:43 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:43.194 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
May 04 10:56:44 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:44.562 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [3,439] milliseconds
May 04 10:56:44 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:44.774 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
May 04 10:56:44 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:44.775 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.30]
May 04 10:56:44 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:44.797 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
May 04 10:56:57 scotty.home.petersen20.de server[638582]: 04-May-2021 10:56:57.060 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
May 04 10:56:57 scotty.home.petersen20.de server[638582]: WARNING: An illegal reflective access operation has occurred
May 04 10:56:57 scotty.home.petersen20.de server[638582]: WARNING: Illegal reflective access by org.candlepin.pki.impl.JSSProviderLoader (file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/) to field java.lang.ClassLoader.usr_paths
May 04 10:56:57 scotty.home.petersen20.de server[638582]: WARNING: Please consider reporting this to the maintainers of org.candlepin.pki.impl.JSSProviderLoader
May 04 10:56:57 scotty.home.petersen20.de server[638582]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
May 04 10:56:57 scotty.home.petersen20.de server[638582]: WARNING: All illegal access operations will be denied in a future release
May 04 10:57:06 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:06.235 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.OwnerCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:06 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:06.488 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:06 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:06.489 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:06 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:06.490 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:06 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:06.698 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:06 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:06.766 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:06 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:06.768 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:07 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:07.107 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.CdnCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:07 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:07.175 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:07 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:07.558 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:07 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:07.559 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:07 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:07.624 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ContentCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:08 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:08.110 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55571f5e]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
May 04 10:57:29 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:29.901 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
May 04 10:57:29 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:29.911 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
May 04 10:57:30 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:30.123 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
May 04 10:57:30 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:30.127 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->1hgf027ahf5c9ch1bx3l4o|4bc1bcb8]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
May 04 10:57:30 scotty.home.petersen20.de server[638582]:  java.base@11.0.11/java.lang.Object.wait(Native Method)
May 04 10:57:30 scotty.home.petersen20.de server[638582]:  java.base@11.0.11/java.util.TimerThread.mainLoop(Timer.java:553)
May 04 10:57:30 scotty.home.petersen20.de server[638582]:  java.base@11.0.11/java.util.TimerThread.run(Timer.java:506)
May 04 10:57:30 scotty.home.petersen20.de server[638582]: 04-May-2021 10:57:30.130 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->1hgf027ahf5c9ch1bx3l4o|4bc1bcb8]-HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

Unfortunalety I cannot interpret this messages ;-(
Perhaps some one can give a hint ?
Best regards

Does starting tomcat manually result in an error?
Do you see any additional errors in the any of the tomcat logs?

Starting tomcat manually on the cli gave no error:

[root@scotty tomcat]# tomcat start
[root@scotty tomcat]# pwd
/var/log/tomcat
[root@scotty tomcat]#

Manual start leeds to the same error entries in the log (see above).
In the localhost.log is this entry:

04-May-2021 14:13:00.284 SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.candlepin.guice.CandlepinContextListener]
java.lang.NoSuchFieldError: SERVER_SENT_EVENTS_TYPE

After the tomcat is started, this message repeatedly every minute:

May 04 14:47:05 scotty.home.petersen20.de server[671974]: Exception in thread "Thread-57 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@2a505beb)" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at ch.qos.logback.classic.Logger.log(Logger.java:765)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at org.jboss.logging.Slf4jLocationAwareLogger.doLog(Slf4jLocationAwareLogger.java:89)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at org.jboss.logging.Slf4jLocationAwareLogger.doLog(Slf4jLocationAwareLogger.java:75)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at org.jboss.logging.Logger.warn(Logger.java:1236)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:47)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
May 04 14:47:05 scotty.home.petersen20.de server[671974]:         at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

This makes me wonder if it installed correctly. Can you share the output of:

rpm -qv tomcatjss
rpm -qV tomcatjss
rpm -qv candlepin
rpm -qV candlepin
[mpetersen@scotty ~]$ rpm -qv tomcatjss
tomcatjss-7.5.0-1.module+el8.3.0+7355+c59bcbd9.noarch
[mpetersen@scotty ~]$ rpm -qV tomcatjss
[mpetersen@scotty ~]$ rpm -qv candlepin
candlepin-3.2.11-1.el8.noarch
[mpetersen@scotty ~]$ rpm -qV candlepin
S.?....T.  c /etc/candlepin/candlepin.conf
.M...UG..  g /etc/candlepin/certs/candlepin-ca.crt
[mpetersen@scotty ~]$

In the installation docs is mentioned for centos 8 , that ruby 2.7 must be enabled. Is this also true for rhel 8 ?

In nightly it is, but since you’re on 4.0 it shouldn’t be. Also, Candlepin is Java based and doesn’t use Ruby. That shouldn’t affect it.

However, my knowledge of Java is lacking here. You may want to verify your JRE is correct. It should be using version 11.

My java version is 11 and the tomcat is 9.0.30 .