Requirements for foreman katello

Priya1 · December 24, 2024, 12:08pm

Problem: We have 3 machines. 1st machine is for foreman UI, 2nd machine is is for Database and 3 machine is for smart proxy. Please help me with the exact steps or docs for foreman-katello installation which you have followed and working fine in production environment. We have done the similar setup but still its not working perfectly and creating lot of issues.

**Expected outcome:**successfully installing foreman katello on the server

**Foreman and Proxy versions:**Foreman 3.12 katello 4.14

Foreman and Proxy plugin versions:

Distribution and version: Rocky 9

Other relevant data:

gvde · December 24, 2024, 12:39pm

Docs are

https://docs.theforeman.org/3.12/Installing_Server/index-katello.html
https://docs.theforeman.org/3.12/Installing_Proxy/index-katello.html

Those work. You’ll really have to post the exact commands you have used to set up your environments and which issues you have.

Priya1 · December 30, 2024, 1:14pm

We have tried to do the below command but we are encountering the error as follows:

foreman-installer --scenario katello
–foreman-db-host dbservername
–foreman-db-password passwd
–foreman-db-database foreman_preprod
–foreman-db-manage false
–katello-candlepin-db-host dbservername
–katello-candlepin-db-name candlepin
–katello-candlepin-db-password passwd
–katello-candlepin-manage-db false
–foreman-proxy-content-pulpcore-manage-postgresql false
–foreman-proxy-content-pulpcore-postgresql-host dbservername
–foreman-proxy-content-pulpcore-postgresql-db-name pulpcore
–foreman-proxy-content-pulpcore-postgresql-password passwd

ERROR:

Error 1: Puppet Private_key resource ‘/etc/candlepin/certs/candlepin-ca.key’ failed. Logs:
/Stage[main]/Certs::Candlepin/Certs::Keypair[katello-default-ca]/Private_key[/etc/candlepin/certs/candlepin-ca.key]/before
before to File[/etc/candlepin/certs/candlepin-ca.key]
/Stage[main]/Certs::Candlepin/Certs::Keypair[katello-default-ca]/Private_key[/etc/candlepin/certs/candlepin-ca.key]
Adding autorequire relationship with File[/root/ssl-build/katello-default-ca.pwd]
Skipping automatic relationship with File[/etc/candlepin/certs/candlepin-ca.key]
Starting to evaluate the resource (489 of 1134)
Could not evaluate: Execution of ‘/bin/openssl rsa -in /root/ssl-build/katello-default-ca.key -passin file:/root/ssl-build/katello-default-ca.pwd -text’ returned 1: Could not find private key from /root/ssl-build/katello-default-ca.key
00EE9A5EDB7F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:
00EE9A5EDB7F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:107:
00EE9A5EDB7F0000:error:11800074:PKCS12 routines:PKCS12_pbe_crypt_ex:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:84:maybe wrong password
Evaluated in 0.01 seconds
Error 2: Puppet Exec resource ‘pulpcore-manager migrate --noinput’ failed. Logs:
/Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]
Adding autorequire relationship with File[/var/lib/pulp]
Adding autorequire relationship with User[pulp]
Starting to evaluate the resource (988 of 1140)
Evaluated in 2.62 seconds
Execpulpcore-manager migrate --noinput
Executing ‘pulpcore-manager migrate --noinput’
/Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns
Traceback (most recent call last):
File “/usr/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 289, in ensure_connection
self.connect()
File “/usr/lib/python3.11/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 270, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py”, line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/psycopg/connection.py”, line 119, in connect
raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: password authentication failed for user “pulp”
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/bin/pulpcore-manager”, line 33, in
sys.exit(load_entry_point(‘pulpcore==3.49.22’, ‘console_scripts’, ‘pulpcore-manager’)())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/lib/python3.11/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
utility.execute()
File “/usr/lib/python3.11/site-packages/django/core/management/init.py”, line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/usr/lib/python3.11/site-packages/django/core/management/base.py”, line 412, in run_from_argv
self.execute(*args, **cmd_options)
File “/usr/lib/python3.11/site-packages/django/core/management/base.py”, line 458, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/core/management/base.py”, line 106, in wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py”, line 117, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/migrations/executor.py”, line 18, in init
self.loader = MigrationLoader(self.connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/migrations/loader.py”, line 58, in init
self.build_graph()
File “/usr/lib/python3.11/site-packages/django/db/migrations/loader.py”, line 235, in build_graph
self.applied_migrations = recorder.applied_migrations()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/migrations/recorder.py”, line 81, in applied_migrations
if self.has_table():
^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/migrations/recorder.py”, line 57, in has_table
with self.connection.cursor() as cursor:
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 330, in cursor
return self._cursor()
^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 306, in _cursor
self.ensure_connection()
File “/usr/lib/python3.11/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 288, in ensure_connection
with self.wrap_database_errors:
File “/usr/lib/python3.11/site-packages/django/db/utils.py”, line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/usr/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 289, in ensure_connection
self.connect()
File “/usr/lib/python3.11/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 270, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py”, line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/psycopg/connection.py”, line 119, in connect
raise last_ex.with_traceback(None)
django.db.utils.OperationalError: connection failed: password authentication failed for user “pulp”
change from ‘notrun’ to [‘0’] failed: ‘pulpcore-manager migrate --noinput’ returned 1 instead of one of [0]

2 errors were detected.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

Priya1 · December 30, 2024, 2:09pm

We fixed the error. Now we are facing the below error. Please help us on this.

Error 1: Puppet Private_key resource ‘/etc/candlepin/certs/candlepin-ca.key’ failed. Logs:
/Stage[main]/Certs::Candlepin/Certs::Keypair[katello-default-ca]/Private_key[/etc/candlepin/certs/candlepin-ca.key]/before
before to File[/etc/candlepin/certs/candlepin-ca.key]
/Stage[main]/Certs::Candlepin/Certs::Keypair[katello-default-ca]/Private_key[/etc/candlepin/certs/candlepin-ca.key]
Adding autorequire relationship with File[/root/ssl-build/katello-default-ca.pwd]
Skipping automatic relationship with File[/etc/candlepin/certs/candlepin-ca.key]
Starting to evaluate the resource (489 of 1134)
Could not evaluate: Execution of ‘/bin/openssl rsa -in /root/ssl-build/katello-default-ca.key -passin file:/root/ssl-build/katello-default-ca.pwd -text’ returned 1: Could not find private key from /root/ssl-build/katello-default-ca.key
00CEBDB9DC7F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:
00CEBDB9DC7F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:107:
00CEBDB9DC7F0000:error:11800074:PKCS12 routines:PKCS12_pbe_crypt_ex:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:84:maybe wrong password
Evaluated in 0.01 seconds

gvde · December 30, 2024, 2:22pm

How exactly? What did you change?

It looks like the ssl-build directory contains some incorrect files.

Which files do you have in /root/ssl-build?

# ls -la /root/ssl-build/
total 188
drwx------.  7 root root  4096 Sep 27 07:25 .
dr-xr-x---. 16 root root  4096 Dec 30 09:41 ..
drwx------.  2 root root  8192 Jun 13  2024 foreman8-content.example.com
drwxr-x---.  2 root root  4096 Mar 20  2024 foreman8.example.com
drwx------.  2 root root 12288 Jun 13  2024 foreman8-puppet.example.com
drwxr-x---.  2 root root  4096 Sep 23 16:42 foreman9-content.cloud.example.com
-rw-r--r--.  1 root root   151 Sep 23 16:42 index.txt
-rw-r--r--.  1 root root    21 Sep 23 16:42 index.txt.attr
-rw-r--r--.  1 root root    21 Sep 23 16:42 index.txt.attr.old
-rw-------.  1 root root  2606 Jul 13  2022 katello-ca-openssl.cnf
-rw-------.  1 root root  2607 Jul 13  2022 katello-ca-openssl.cnf.1
-rw-r--r--.  1 root root  8336 Jul 13  2022 katello-default-ca-1.0-1.noarch.rpm
-rw-r--r--.  1 root root  8926 Jul 13  2022 katello-default-ca-1.0-1.src.rpm
-rw-r--r--.  1 root root  2524 Jul 13  2022 katello-default-ca.crt
-rw-------.  1 root root  3434 Jul 13  2022 katello-default-ca.key
-r--------.  1 root root    24 Mar 12  2024 katello-default-ca.pwd
-rw-r--r--.  1 root root 10620 Jul 13  2022 katello-server-ca-1.0-1.noarch.rpm
-rw-r--r--.  1 root root 11232 Jul 13  2022 katello-server-ca-1.0-1.src.rpm
-rw-r--r--.  1 root root 10620 Apr  7  2023 katello-server-ca-1.0-2.noarch.rpm
-rw-r--r--.  1 root root 11225 Apr  7  2023 katello-server-ca-1.0-2.src.rpm
-rw-r--r--.  1 root root 10620 Mar 20  2024 katello-server-ca-1.0-3.noarch.rpm
-rw-r--r--.  1 root root 11231 Mar 20  2024 katello-server-ca-1.0-3.src.rpm
-rw-r--r--.  1 root root  6189 Sep 27 07:25 katello-server-ca.crt
lrwxrwxrwx.  1 root root    37 Jul 13  2022 KATELLO-TRUSTED-SSL-CERT -> /root/ssl-build/katello-server-ca.crt
-rw-r--r--.  1 root root    89 Mar 20  2024 latest.txt
drwx------.  2 root root   225 Jul 13  2022 localhost
-rw-r--r--.  1 root root    41 Sep 23 16:42 serial

katello-default-ca.pwd should contain a password for the private key in katello-default-ca.key, i.e.

# openssl rsa -in /root/ssl-build/katello-default-ca.key -passin file:/root/ssl-build/katello-default-ca.pwd -noout -modulus
Modulus=B44A7B1351DDF5155590D13F33F5711483956...

should work without errors.

Did you ever run foreman-installer before on that server with different arguments? It looks like you have tried custom certificates before…

Priya1 · December 30, 2024, 2:40pm

Thanks @gvde, but still its not working …

[root@servername~]# openssl rsa -in /root/ssl-build/katello-default-ca.key -passin file:/root/ssl-build/katello-default-ca.pwd -noout -modulus

Could not find private key from /root/ssl-build/katello-default-ca.key
007E2E55317F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:
007E2E55317F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:107:
007E2E55317F0000:error:11800074:PKCS12 routines:PKCS12_pbe_crypt_ex:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:84:maybe wrong password

contents of /root/ssl-build:

[root@servername~]# cd /root/ssl-build
[root@servername ssl-build]# ls -ll
total 60
-rw-r-----. 1 root root 156 Dec 30 08:14 index.txt
-rw-r-----. 1 root root 21 Dec 30 08:14 index.txt.attr
-rw-r-----. 1 root root 21 Dec 30 08:14 index.txt.attr.old
-rw-------. 1 root root 2610 Dec 30 08:14 katello-ca-openssl.cnf
-rw-------. 1 root root 2611 Dec 30 08:14 katello-ca-openssl.cnf.1
-rw-r-----. 1 root root 1237 Dec 30 14:29 katello-default-ca.crt
-rw-------. 1 root root 1886 Dec 30 14:28 katello-default-ca.key
-rw-r–r–. 1 root root 2508 Dec 30 08:14 katello-default-ca_old.crt
-rw-r–r–. 1 root root 3446 Dec 30 08:14 katello-default-ca_old.key
-r--------. 1 root root 24 Dec 30 11:27 katello-default-ca_old.pwd
-r--------. 1 root root 24 Dec 30 14:34 katello-default-ca.pwd
-rw-r–r–. 1 root root 1237 Dec 30 14:34 katello-server-ca.crt
lrwxrwxrwx. 1 root root 37 Dec 30 08:14 KATELLO-TRUSTED-SSL-CERT → /root/ssl-build/katello-server-ca.crt
-rw-r–r–. 1 root root 23 Dec 30 08:14 latest.txt
drwx------. 2 root root 128 Dec 30 08:14 localhost
-rw-r-----. 1 root root 41 Dec 30 08:14 serial
drwx------. 2 root root 4096 Dec 30 08:14 servername.com

Priya1 · December 30, 2024, 2:41pm

We didnot do any custom certificates and this is the fresh machine with fresh installation

gvde · December 30, 2024, 3:56pm

It looks as if you ran foreman-installer a first time at 08:14 and then again at 14:28, resetting the certificates somehow. The password in the .pwd file doesn’t match the password of the private key. It’s also strange that the size of katello-default-ca.key is much smaller than katello-default-ca_old.key which matches the size I see on our foreman server.

Can you check the content of the cert and key files and compare it to the _old files?

# cat katello-default-ca.crt
-----BEGIN CERTIFICATE-----
MIIHGjCCB...
...
-----END CERTIFICATE-----
# cat katello-default-ca.key 
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIJrTBXB...
...
-----END ENCRYPTED PRIVATE KEY-----
# openssl x509 -in katello-default-ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            06:44:a2:b4:c8:36:cb:04:d5:5e:2b:3f:83:cb:94:56:c1:89:15:19
        Signature Algorithm: sha256WithRSAEncryption
...

I guess your files contain something different or shorter keys for some reason. It’s unclear to me how it’s possible that the private key has been encrypted with a different password. You could try to decrypt the key using _old pwd file:

# openssl rsa -in /root/ssl-build/katello-default-ca.key -passin file:/root/ssl-build/katello-default-ca_old.pwd -noout -modulus

and also check the cert and key in _old:

# openssl rsa -in /root/ssl-build/katello-default-ca_old.key -passin file:/root/ssl-build/katello-default-ca_old.pwd -noout -modulus
...
# openssl x509 -in /root/ssl-build/katello-default-ca_old.crt -noout -text
...

If the _old files contain the correct hostname and the _old.pwd password decrypts the you could try to move the katello-default-ca.* files away and rename the katello-default-ca_old.* files to katello-default-ca.* so that it uses those “old” files. I think you’ll have to run foreman-installer --certs-update-all after that to get it properly deployed.

Otherwise, one of the developers would have to chime in. I don’t know how you have got those _old files and how it’s possible to get a broken katello-default-ca.* file set there.

Sidenote: please put your text output into preformatted blocks. They are much easier to read and have own scrollbars, if it’s longer.

Please also use example.{com,edu,net,org} domain names when hiding your own host/domain name. It’s not nice to hide your domain name from the public and instead using a name which is registered to someone else. (unless you own servername dot com).

Priya1 · December 31, 2024, 11:13am

We tried removing /root/ssl-build and ran foreman-installer again. Worked finally… Now we are getting another error when run “foreman-maintain health check”.

foreman-maintain health check
Running preparation steps required to run the next scenarios

Check whether all services are running: [OK]

Running ForemanMaintain::Scenario::FilteredScenario

Check number of fact names in database: [OK]

Check whether all services are running: [OK]

Check whether all services are running using the ping call: [FAIL]
Some components are failing: candlepin, candlepin_auth, candlepin_events

Continue with step [Restart applicable services]?, [y(yes), n(no)] y
Restart applicable services:

Restarting the following service(s):
postgresql (candlepin), tomcat
\ restarting postgresql (candlepin)
postgresql (candlepin) is remote and is UP. Remote databases are not managed by foreman-maintain and therefore was not restarted.

| All services restarted
/ Try 1/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events

Try 2/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events
\ Try 3/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events
| Try 4/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events
/ Try 5/5: checking status of hammer ping [FAIL]
Server response check failed!

Rerunning the check after fix procedure
Check whether all services are running using the ping call: [FAIL]
Some components are failing: candlepin, candlepin_auth, candlepin_events

Continue with step [Restart applicable services]?, [y(yes), n(no)] n
Check for paused tasks: [OK]

Check to verify no empty CA cert requests exist: [OK]

Scenario [ForemanMaintain::Scenario::FilteredScenario] failed.

The following steps ended up in failing state:

[server-ping]

Resolve the failed steps and rerun the command.
In case the failures are false positives, use
–whitelist=“server-ping”

We tried executing hammer ping to check the status:

hammer ping

database:
Status: ok
Server Response: Duration: 0ms
cache:
servers:
1) Status: ok
Server Response: Duration: 0ms
candlepin:
Status: FAIL
Server Response: Message: Failed to open TCP connection to localhost:23443 (Connection refused - connect(2) for “localhost” port 23443)
candlepin_auth:
Status: FAIL
Server Response: Message: A backend service [ Candlepin ] is unreachable
candlepin_events:
Status: FAIL
message: Not running
Server Response: Duration: 0ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
pulp3:
Status: ok
Server Response: Duration: 43ms
pulp3_content:
Status: ok
Server Response: Duration: 40ms
foreman_tasks:
Status: ok
Server Response: Duration: 2ms

We have allowed firewall for the port 23443 but still Candlepin service is not getting started.

Also we could see that tomcat is running on the server.

systemctl status tomcat
● tomcat.service - Apache Tomcat Web Application Container
Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; preset: disabled)
Active: active (running) since Tue 2024-12-31 11:07:57 UTC; 2min 26s ago
Main PID: 775161 (java)
Tasks: 37 (limit: 407992)
Memory: 940.6M
CPU: 26.759s
CGroup: /system.slice/tomcat.service
└─775161 /usr/lib/jvm/jre-17/bin/java -Xms1024m -Xmx4096m -Dcom.redhat.fips=false -Djava.security.auth.login.config=/usr/share/tomcat/conf/logi>

could you please help us on this issue.

gvde · December 31, 2024, 11:45am

Again:

Please post the system log/output properly. They are really very hard to read if you don’t format it properly. Really. Believe me. A little effort on your side makes it so much easier for everyone else who wants to help you.

Why would you do that? That doesn’t make any sense and random changes like this are those which make problems in the long run because noone would expect that in a normal setup.

Undo that change. It’s absolutely pointless to open a firewall port to the public which isn’t open to the public anyway because it is only listening and being used on localhost/127.0.0.1.

Which other changes did you try which you did not write about?

You should check the candlepin logs.

Priya1 · January 2, 2025, 10:28am

Please find the candlepin logs:

cat candlepin.log
2025-01-02 10:02:37,771 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-02 10:02:37,778 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-02 10:02:37,782 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-02 10:02:37,825 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-02 10:02:37,834 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-02 10:02:37,839 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-02 10:02:37,908 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-02 10:02:39,419 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-02 10:02:39,569 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-02 10:02:39,920 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-02 10:02:39,920 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-02 10:02:39,920 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-02 10:02:39,922 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-02 10:02:39,922 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-02 10:02:39,922 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-02 10:02:39,925 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-02 10:02:39,925 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-02 10:02:39,925 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-02 10:02:39,926 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-02 10:02:39,926 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-02 10:02:41,594 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:02:41,712 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:02:41,713 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:02:42,571 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381
2025-01-02 10:07:26,602 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-02 10:07:26,603 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-02 10:07:26,607 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-02 10:07:26,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-02 10:07:26,656 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-02 10:07:26,660 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-02 10:07:26,727 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-02 10:07:28,161 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-02 10:07:28,310 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-02 10:07:28,645 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-02 10:07:28,646 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-02 10:07:28,646 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-02 10:07:28,646 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-02 10:07:28,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-02 10:07:28,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-02 10:07:28,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-02 10:07:28,650 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-02 10:07:28,650 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-02 10:07:28,650 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-02 10:07:28,652 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-02 10:07:30,296 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:31,257 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381


cat /var/log/candlepin/error.log
2025-01-02 10:02:41,594 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:02:41,712 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:02:41,713 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:30,296 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:08:43,715 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:08:43,828 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:08:43,828 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute

Please let me know if any other details required from our end.

Priya1 · January 2, 2025, 1:43pm

We found this error in tomcat logs:

Dec 30 08:15:15  server[3376632]: 30-Dec-2024 08:15:15.456 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
Dec 30 08:15:15  server[3376632]: 30-Dec-2024 08:15:15.460 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors

Priya1 · January 3, 2025, 2:01pm

Can someone please help us.

gvde · January 3, 2025, 2:17pm

The tomcat error is from Dec 30 and the candlepin log is from Jan 02. As the tomcat log message says: the full details are in the container log.

I would suggest you stop the tomcat.service

# systemctl stop tomcat.service

Make sure that no other/old tomcat process is running:

# ps -ef | grep tomcat
...
# lsof /var/log/candlepin/candlepin.log
...

Then start tomcat.service again and post the logs from tomcat.service with the corresponding container logs in /var/log/candlepin.

Priya1 · January 6, 2025, 6:27am

cat /var/log/tomcat/catalina.2025-01-06.log

06-Jan-2025 06:10:47.786 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 06:10:47.786 INFO [Thread-5] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
06-Jan-2025 06:10:47.787 INFO [Thread-5] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 06:10:47.790 INFO [Thread-5] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 06:10:48.068 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
06-Jan-2025 06:10:48.235 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
06-Jan-2025 06:10:48.337 WARNING [main] org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the [ciphers] attribute in a manner consistent with the latest OpenSSL development branch. Some of the specified [ciphers] are not supported by the configured SSL engine for this connector (which may use JSSE or an older OpenSSL version) and have been skipped: [[TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256]]
06-Jan-2025 06:10:48.338 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["https-jsse-nio-127.0.0.1-23443"]]
        org.apache.catalina.LifecycleException: Protocol handler initialization failed
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1011)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardService.initInternal(StandardService.java:554)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1046)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:686)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:709)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:569)
                at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
        Caused by: java.lang.IllegalArgumentException: /etc/candlepin/certs/keystore (Permission denied)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:115)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:228)
                at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1334)
                at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1347)
                at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:654)
                at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:75)
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1009)
                ... 13 more
        Caused by: java.io.FileNotFoundException: /etc/candlepin/certs/keystore (Permission denied)
                at java.base/java.io.FileInputStream.open0(Native Method)
                at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
                at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
                at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
                at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
                at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
                at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:118)
                at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:210)
                at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:254)
                at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:308)
                at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:268)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:113)
                ... 20 more
06-Jan-2025 06:10:48.339 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [388] milliseconds
06-Jan-2025 06:10:48.360 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
06-Jan-2025 06:10:48.360 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
06-Jan-2025 06:10:48.364 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
06-Jan-2025 06:10:51.210 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
06-Jan-2025 06:10:52.832 INFO [main] liquibase.database.null Set default schema name to public
06-Jan-2025 06:10:52.850 INFO [main] liquibase.changelog.null Reading from public.databasechangelog
06-Jan-2025 06:10:53.533 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.549 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.552 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.OwnerCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.563 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.564 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.564 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.573 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.573 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.594 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.621 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:53.621 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@70c8e284]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
06-Jan-2025 06:10:56.248 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
06-Jan-2025 06:10:56.251 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
06-Jan-2025 06:10:56.254 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jan-2025 06:10:56.255 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8g5pxt3zwke43|7d23abc1]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 java.base@17.0.13/java.util.TimerThread.mainLoop(Timer.java:563)
 java.base@17.0.13/java.util.TimerThread.run(Timer.java:516)
06-Jan-2025 06:10:56.255 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8g5pxt3zwke43|7d23abc1]-HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 06:10:56.256 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8g5pxt3zwke43|7d23abc1]-HelperThread-#1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 06:10:56.256 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [candlepin] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8g5pxt3zwke43|7d23abc1]-HelperThread-#2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.13/java.lang.Object.wait(Native Method)
 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
06-Jan-2025 06:10:56.256 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5e9fe0c0]) and a value of type [liquibase.SingletonScopeManager] (value [liquibase.SingletonScopeManager@3387e8e0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 06:10:56.256 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3070cbe]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(1463535808<open>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Jan-2025 06:10:56.260 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/candlepin] has finished in [7,896] ms
06-Jan-2025 06:10:56.262 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [7923] milliseconds
06-Jan-2025 06:15:54.764 INFO [C3P0PooledConnectionPoolManager[identityToken->2sykj4b8g5pxt3zwke43|7d23abc1]-AdminTaskTimer] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1349)
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1337)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1174)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1141)
                at com.mchange.v2.resourcepool.BasicResourcePool.checkIdleResources(BasicResourcePool.java:1673)
                at com.mchange.v2.resourcepool.BasicResourcePool.access$2000(BasicResourcePool.java:44)
                at com.mchange.v2.resourcepool.BasicResourcePool$CheckIdleResourcesTask.run(BasicResourcePool.java:2214)
                at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
                at java.base/java.util.TimerThread.run(Timer.java:516)




cat candlepin.log

2025-01-06 06:10:51,372 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-06 06:10:51,377 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-06 06:10:51,382 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-06 06:10:51,423 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-06 06:10:51,431 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-06 06:10:51,435 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-06 06:10:51,500 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-06 06:10:52,890 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-06 06:10:53,040 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-06 06:10:53,385 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-06 06:10:53,385 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-06 06:10:53,386 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-06 06:10:53,386 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-06 06:10:53,386 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-06 06:10:53,386 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-06 06:10:53,387 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-06 06:10:53,387 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-06 06:10:53,387 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-06 06:10:53,388 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-06 06:10:53,388 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-06 06:10:53,388 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-06 06:10:53,388 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-06 06:10:53,389 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-06 06:10:53,389 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-06 06:10:53,389 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-06 06:10:53,390 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-06 06:10:53,390 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-06 06:10:53,390 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-06 06:10:53,390 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-06 06:10:53,391 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-06 06:10:53,391 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-06 06:10:53,391 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-06 06:10:55,041 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-06 06:10:55,155 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-06 06:10:55,155 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-06 06:10:55,988 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381


Please let us know any other logs are required for further troubleshooting.

gvde · January 6, 2025, 8:26am

Candlepin cannot read the keystore file. What do you have in that directory?

# ls -la /etc/candlepin/certs/
total 36
drwxr-xr-x. 3 root root     166 Nov 19 19:02 .
drwxr-xr-x. 3 root root      59 Nov 19 19:02 ..
-r--r-----. 1 root tomcat  2524 Jul 13  2022 candlepin-ca.crt
-r--r-----. 1 root tomcat 11151 Oct 20 16:55 candlepin-ca.key
-rw-r-----. 1 root tomcat  4663 Jan 16  2024 keystore
-r--r-----. 1 root tomcat    32 Nov 19 19:02 keystore_password-file
-rw-r-----. 1 root tomcat  4050 Jan 16  2024 truststore
-r--r-----. 1 root tomcat    32 Nov 19 19:02 truststore_password-file
drwxr-xr-x. 2 root root      37 Oct 13  2023 upstream

Priya1 · January 6, 2025, 8:32am

Output of ls -la under /etc/candlepin/certs


 ls -la
total 60
drwxr-xr--. 3 root   root    4096 Jan  3 12:27 .
drwxr-xr-x. 3 root   root      89 Jan  3 12:21 ..
-r--r-----. 1 root   tomcat  2508 Jan  3 12:27 candlepin-ca.crt
-r--r-----. 1 root   tomcat 11148 Jan  3 12:27 candlepin-ca.key
-rw-r--r--. 1 tomcat tomcat  4808 Jan  3 12:27 keystore
-r--r-----. 1 root   tomcat    32 Jan  3 12:20 keystore_password-file
-rw-r-----. 1 root   tomcat  4166 Jan  3 12:27 truststore
-r--r-----. 1 root   tomcat    32 Jan  3 12:20 truststore_password-file
drw-r--r--. 2 root   root      37 Jan  3 12:20 upstream

gvde · January 6, 2025, 8:53am

That is odd. foreman-installer should set those permissions the way I have them:

    keystore { $keystore:
      ensure        => present,
      password_file => $keystore_password_path,
      owner         => 'root',
      group         => $group,
      mode          => '0640',
    }

Can you run foreman-installer again and check if it changes the permissions? Also check the installer log katello.log for details if it handles the keystore file or not.

Are the selinux correct?

# ls -laZ /etc/candlepin/certs/
total 36
drwxr-xr-x. 3 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0          166 Nov 19 19:02 .
drwxr-xr-x. 3 root root   system_u:object_r:candlepin_etc_rw_t:s0                 59 Nov 19 19:02 ..
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0  2524 Jul 13  2022 candlepin-ca.crt
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0 11151 Oct 20 16:55 candlepin-ca.key
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4663 Jan 16  2024 keystore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Nov 19 19:02 keystore_password-file
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4050 Jan 16  2024 truststore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Nov 19 19:02 truststore_password-file
drwxr-xr-x. 2 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0           37 Oct 13  2023 upstream

Priya1 · January 6, 2025, 9:41am

We ran the foreman-installer command again and below are the output of /etc/candlepin/certs/

ls -laZ /etc/candlepin/certs/
total 60
drwxr-xr--. 3 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0         4096 Jan  3 12:27 .
drwxr-xr-x. 3 root root   system_u:object_r:candlepin_etc_rw_t:s0                 89 Jan  3 12:21 ..
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0  2508 Jan  3 12:27 candlepin-ca.crt
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_ca_cert_r_t:s0 11148 Jan  3 12:27 candlepin-ca.key
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4808 Jan  3 12:27 keystore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Jan  3 12:20 keystore_password-file
-rw-r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0         4166 Jan  3 12:27 truststore
-r--r-----. 1 root tomcat system_u:object_r:candlepin_etc_certs_rw_t:s0           32 Jan  3 12:20 truststore_password-file
drw-r--r--. 2 root root   system_u:object_r:candlepin_etc_certs_rw_t:s0           37 Jan  3 12:20 upstream

cat /var/log/foreman-installer/katello.log | grep -i keystore
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore_password-file]/seluser: Found seluser default 'system_u' for /etc/candlepin/certs/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore_password-file]/selrole: Found selrole default 'object_r' for /etc/candlepin/certs/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore_password-file]/seltype: Found seltype default 'candlepin_etc_certs_rw_t' for /etc/candlepin/certs/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore_password-file]/selrange: Found selrange default 's0' for /etc/candlepin/certs/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/pki/katello/keystore_password-file]/seluser: Found seluser default 'system_u' for /etc/pki/katello/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/pki/katello/keystore_password-file]/selrole: Found selrole default 'object_r' for /etc/pki/katello/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/pki/katello/keystore_password-file]/seltype: Found seltype default 'cert_t' for /etc/pki/katello/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/pki/katello/keystore_password-file]/selrange: Found selrange default 's0' for /etc/pki/katello/keystore_password-file
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore]/seluser: Found seluser default 'system_u' for /etc/candlepin/certs/keystore
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore]/selrole: Found selrole default 'object_r' for /etc/candlepin/certs/keystore
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore]/seltype: Found seltype default 'candlepin_etc_certs_rw_t' for /etc/candlepin/certs/keystore
2025-01-06 08:59:25 [DEBUG ] [configure] /File[/etc/candlepin/certs/keystore]/selrange: Found selrange default 's0' for /etc/candlepin/certs/keystore
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore[/etc/candlepin/certs/keystore]/before: before to File[/etc/candlepin/certs/keystore]
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore_password-file]: Adding autorequire relationship with Group[tomcat]
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore[/etc/candlepin/certs/keystore]: Adding autorequire relationship with File[/etc/candlepin/certs/keystore_password-file]
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore]: Adding autorequire relationship with Group[tomcat]
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore_certificate[/etc/candlepin/certs/keystore:tomcat]: Adding autorequire relationship with File[/etc/candlepin/certs/keystore_password-file]
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore_certificate[/etc/candlepin/certs/keystore:tomcat]: Adding autorequire relationship with File[/etc/candlepin/certs/candlepin-ca.crt]
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore_certificate[/etc/candlepin/certs/keystore:tomcat]: Adding autorequire relationship with File[/etc/candlepin/certs/keystore]
2025-01-06 08:59:25 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/pki/katello/keystore_password-file]: Adding autorequire relationship with File[/etc/pki/katello]
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore_password-file]: Starting to evaluate the resource (163 of 1132)
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore_password-file]: Evaluated in 0.00 seconds
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore[/etc/candlepin/certs/keystore]: Starting to evaluate the resource (164 of 1132)
2025-01-06 08:59:26 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/keystore -storepass:file /etc/candlepin/certs/keystore_password-file -J-Dcom.redhat.fips=false'
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore[/etc/candlepin/certs/keystore]: Evaluated in 0.28 seconds
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore]: Starting to evaluate the resource (165 of 1132)
2025-01-06 08:59:26 [INFO  ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore]/owner: owner changed 'tomcat' to 'root'
2025-01-06 08:59:26 [INFO  ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore]/mode: mode changed '0655' to '0640'
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore]: The container Class[Certs::Candlepin] will propagate my refresh event
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore]: The container Class[Certs::Candlepin] will propagate my refresh event
2025-01-06 08:59:26 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/candlepin/certs/keystore]: Evaluated in 0.00 seconds
2025-01-06 08:59:26 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/truststore -storepass:file /etc/candlepin/certs/truststore_password-file -J-Dcom.redhat.fips=false'
2025-01-06 08:59:27 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/pki/katello/keystore_password-file]: Starting to evaluate the resource (217 of 1132)
2025-01-06 08:59:27 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/pki/katello/keystore_password-file]: Nothing to manage: no ensure and the resource doesn't exist
2025-01-06 08:59:27 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/File[/etc/pki/katello/keystore_password-file]: Evaluated in 0.00 seconds
2025-01-06 08:59:27 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/truststore -storepass:file /etc/candlepin/certs/truststore_password-file -alias artemis-client -J-Dcom.redhat.fips=false'
2025-01-06 08:59:27 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/truststore -storepass:file /etc/candlepin/certs/truststore_password-file -alias artemis-client -J-Dcom.redhat.fips=false'
2025-01-06 08:59:28 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore_certificate[/etc/candlepin/certs/keystore:tomcat]: Starting to evaluate the resource (493 of 1134)
2025-01-06 08:59:28 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/keystore -storepass:file /etc/candlepin/certs/keystore_password-file -alias tomcat -J-Dcom.redhat.fips=false'
2025-01-06 08:59:28 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/keystore -storepass:file /etc/candlepin/certs/keystore_password-file -alias tomcat -J-Dcom.redhat.fips=false'
2025-01-06 08:59:28 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Keystore_certificate[/etc/candlepin/certs/keystore:tomcat]: Evaluated in 0.54 seconds
2025-01-06 08:59:28 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/truststore -storepass:file /etc/candlepin/certs/truststore_password-file -alias candlepin-ca -J-Dcom.redhat.fips=false'
2025-01-06 08:59:29 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/truststore -storepass:file /etc/candlepin/certs/truststore_password-file -alias candlepin-ca -J-Dcom.redhat.fips=false'

Priya1 · January 6, 2025, 9:42am

sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

Requirements for foreman katello

foreman-maintain health check Running preparation steps required to run the next scenarios

Check whether all services are running: [OK]

Running ForemanMaintain::Scenario::FilteredScenario

Check number of fact names in database: [OK]

Check whether all services are running: [OK]

Check whether all services are running using the ping call: [FAIL] Some components are failing: candlepin, candlepin_auth, candlepin_events

Rerunning the check after fix procedure Check whether all services are running using the ping call: [FAIL] Some components are failing: candlepin, candlepin_auth, candlepin_events

Continue with step [Restart applicable services]?, [y(yes), n(no)] n Check for paused tasks: [OK]

Check to verify no empty CA cert requests exist: [OK]

foreman-maintain health check
Running preparation steps required to run the next scenarios

Check whether all services are running using the ping call: [FAIL]
Some components are failing: candlepin, candlepin_auth, candlepin_events

Rerunning the check after fix procedure
Check whether all services are running using the ping call: [FAIL]
Some components are failing: candlepin, candlepin_auth, candlepin_events

Continue with step [Restart applicable services]?, [y(yes), n(no)] n
Check for paused tasks: [OK]