Unable install Foreman server (Forman 3.8+katello 4.10)

sarang · December 7, 2023, 10:03am

Problem:
Unable install Forman server (Forman 3.8+katello 4.10)

Expected outcome:
Should be able to install Foreman

Foreman and Proxy versions:
Foreman 3.8 and Katello 4.10
Foreman and Proxy plugin versions:

Distribution and version:
[root@sjprdsatapp02 ~]# rpm -qa | grep -i foreman
foreman-proxy-3.8.0-1.el8.noarch
foreman-installer-3.8.0-2.el8.noarch
foreman-debug-3.8.0-1.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.19-1.fm3_7.el8.noarch
foreman-release-3.8.0-1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.3-1.fm3_7.el8.noarch
rubygem-hammer_cli_foreman-3.8.0-1.el8.noarch
foreman-3.8.0-1.el8.noarch
foreman-service-3.8.0-1.el8.noarch
foreman-postgresql-3.8.0-1.el8.noarch
rubygem-foreman_remote_execution-11.1.1-1.fm3_8.el8.noarch
rubygem-foreman-tasks-8.3.3-1.fm3_8.el8.noarch
foreman-installer-katello-3.8.0-2.el8.noarch
foreman-cli-3.8.0-1.el8.noarch
foreman-dynflow-sidekiq-3.8.0-1.el8.noarch
rubygem-foreman_maintain-1.3.5-1.el8.noarch
[root@sjprdsatapp02 ~]# rpm -qa | grep -i katello
katello-repos-4.10.0-1.el8.noarch
katello-4.10.0-1.el8.noarch
katello-common-4.10.0-1.el8.noarch
rubygem-katello-4.10.0-1.el8.noarch
katello-debug-4.10.0-1.el8.noarch
katello-certs-tools-2.9.0-2.el8.noarch
rubygem-hammer_cli_katello-1.11.0-0.1.pre.master.20230905173948git1fa5f0c.el8.noarch
foreman-installer-katello-3.8.0-2.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch

Other relevant data:
<Steps followed…

Enable the repositories

subscription-manager repos --disable “*”

dnf clean all

dnf install -y https://yum.theforeman.org/releases/3.8/el8/x86_64/foreman-release.rpm

dnf install -y https://yum.theforeman.org/katello/4.10/katello/el8/x86_64/katello-repos-latest.rpm

dnf install -y https://yum.puppet.com/puppet7-release-el-8.noarch.rpm (change http to https in puppet repo baseurl)

Enable the modules

dnf module enable katello:el8 pulpcore:el8 (first try this)

dnf module disable postgresql (only if above does not work)

dnf module enable ruby:2.7 katello:el8 pulpcore:el8 foreman:el8 postgresql:12

Install foreman packages:

dnf update

dnf install foreman-installer-katello

Synchronizing the System Clock With chronyd

dnf install chrony

systemctl enable --now chronyd

Configuring Foreman server:
Basic command:
[root@sjprdsatapp02 ~]# foreman-installer --scenario katello
–foreman-initial-organization “cadence”
–foreman-initial-location “sanjose”
–foreman-initial-admin-username admin
–foreman-initial-admin-password admin123\

Installation looks successfull however, can not login to webui.

[root@sjprdsatapp02 ~]# foreman-installer --scenario katello --foreman-initial-organization "cadence" --foreman-initial-location "sanjose" --foreman-initial-admin-username admin --foreman-initial-admin-password admin123
2023-12-07 01:27:55 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-12-07 01:27:57 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-12-07 01:27:57 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-12-07 01:27:59 [NOTICE] [configure] Starting system configuration.
2023-12-07 01:30:01 [NOTICE] [configure] 250 configuration steps out of 1406 steps complete.
2023-12-07 01:30:39 [NOTICE] [configure] 500 configuration steps out of 1408 steps complete.
2023-12-07 01:31:28 [NOTICE] [configure] 750 configuration steps out of 1413 steps complete.
2023-12-07 01:31:33 [NOTICE] [configure] 1000 configuration steps out of 1436 steps complete.
2023-12-07 01:33:13 [NOTICE] [configure] 1250 configuration steps out of 1436 steps complete.
2023-12-07 01:35:33 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/10: katello:correct_repositories. This may take a long while.
=============================================
Upgrade Step 2/10: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.
=============================================
Upgrade Step 3/10: katello:upgrades:4.0:remove_ostree_puppet_content. =============================================
Upgrade Step 4/10: katello:upgrades:4.1:sync_noarch_content. =============================================
Upgrade Step 5/10: katello:upgrades:4.1:fix_invalid_pools. I, [2023-12-07T01:35:43.099729 #37175]  INFO -- : Corrected 0 invalid pools
I, [2023-12-07T01:35:43.099757 #37175]  INFO -- : Removed 0 orphaned pools
=============================================
Upgrade Step 6/10: katello:upgrades:4.1:reupdate_content_import_export_perms. =============================================
Upgrade Step 7/10: katello:upgrades:4.2:remove_checksum_values. =============================================
Upgrade Step 8/10: katello:upgrades:4.4:publish_import_cvvs. =============================================
Upgrade Step 9/10: katello:upgrades:4.8:fix_incorrect_providers. Fixing incorrect providers
Fixed 0 incorrect providers
Cleaning Candlepin orphaned custom products for organization cadence
Failed upgrade task: katello:upgrades:4.8:fix_incorrect_providers, see logs for more information.
=============================================
Upgrade Step 10/10: katello:upgrades:4.8:regenerate_imported_repository_metadata. No repositories found for regeneration.
  Success!
  * Foreman is running at https://sjprdsatapp02.cadence.com
      Initial credentials are admin / admin123
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  * Foreman Proxy is running at https://sjprdsatapp02.cadence.com:9090

The full log is at /var/log/foreman-installer/katello.log

hammer status indcates failure with candlepin.

[root@sjprdsatapp02 ~]# hammer ping katello
candlepin:
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:
    Status:          FAIL
    Server Response: Message: Katello::Errors::CandlepinNotRunning
candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:
    Status:          ok
    Server Response: Duration: 57ms
pulp3_content:
    Status:          ok
    Server Response: Duration: 55ms
foreman_tasks:
    Status:          FAIL
    Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

Attaching /var/log/foreman-installer/katello.log to the ticket.
katello.log (2.3 MB)

Also attaching error I see on GUI

sarang · December 7, 2023, 11:19am

Let me know if more information is required.
Attaching /var/log/foreman/production.log.
Error seen in production.log

2023-12-07T01:35:43 [I|dyn|] start terminating clock...
2023-12-07T01:35:48 [E|app|57d63162] Error occurred while starting Katello::CandlepinEventListener
2023-12-07T01:35:48 [E|app|57d63162] Connection refused - connect(2) for "localhost" port 61613
2023-12-07T01:35:48 [E|app|57d63162] /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in `initialize'
[production.log|attachment](upload://nXEbls7tx4C7XMJ7GwKyb6ftSDK.log) (192 KB)

production.log (2.3 MB)

jeremylenz · December 7, 2023, 2:49pm

How much RAM does the system have? This seems like the tomcat service may be getting killed, along with some others… let’s make sure the system meets the minimum requirements as outlined in Installing Foreman Server with Katello 4.10 Plugin on CentOS/RHEL

Once you’ve verified that, you may also try installing foreman-maintain and then

foreman-maintain service restart

Then check foreman-maintain service status to make sure all is OK.

sarang · December 7, 2023, 6:18pm

@jeremylenz system has enough RAM.
The system is meeting minimum requirements.
I have tried reinstalling the foreman multiple times but it fails with this same candlepin issue everytime.

[root@sjprdsatapp02 ~]# hostname
sjprdsatapp02.cadence.com
[root@sjprdsatapp02 ~]# hostname -f
sjprdsatapp02.cadence.com

[root@sjprdsatapp02 ~]# free -g
              total        used        free      shared  buff/cache   available
Mem:            754          15         735           0           4         735
Swap:            15           0          15

[root@sjprdsatapp02 ~]# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              32
On-line CPU(s) list: 0-31
Thread(s) per core:  1
Core(s) per socket:  16
Socket(s):           2

[root@sjprdsatapp02 ~]# localectl
   System Locale: LANG=en_US.UTF-8
       VC Keymap: us
      X11 Layout: us

Here’s the service status after running ‘foreman-maintain service restart’
Does not show any issue with tomcat. however, still seeing same error ‘CandlepinNotRunning’ in hammer status.

[root@sjprdsatapp02 ~]# foreman-maintain service restart
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services:

Restarting the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
/ All services restarted                                              [OK]
--------------------------------------------------------------------------------

[root@sjprdsatapp02 ~]# foreman-maintain service status
Running Status Services
================================================================================
Get status of applicable services:

Displaying the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
| displaying redis
● redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis.service.d
           └─90-limits.conf
   Active: active (running) since Thu 2023-12-07 10:14:31 PST; 1min 8s ago
  Process: 57146 ExecStop=/usr/libexec/redis-shutdown (code=exited, status=0/SUCCESS)
 Main PID: 57161 (redis-server)
   Status: "Ready to accept connections"
    Tasks: 5 (limit: 3355442)
   Memory: 9.1M
   CGroup: /system.slice/redis.service
           └─57161 /usr/bin/redis-server 127.0.0.1:6379

Dec 07 10:14:31 sjprdsatapp02.cadence.com systemd[1]: Starting Redis persistent key-value database...
Dec 07 10:14:31 sjprdsatapp02.cadence.com systemd[1]: Started Redis persistent key-value database.
/ displaying postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/postgresql.service.d
           └─postgresql.conf
   Active: active (running) since Thu 2023-12-07 10:14:31 PST; 1min 8s ago
  Process: 57176 ExecStartPre=/usr/libexec/postgresql-check-db-dir postgresql (code=exited, status=0/SUCCESS)
 Main PID: 57179 (postmaster)
    Tasks: 242 (limit: 3355442)
   Memory: 490.2M
   CGroup: /system.slice/postgresql.service
           ├─57179 /usr/bin/postmaster -D /var/lib/pgsql/data
           ├─57180 postgres: logger
           ├─57182 postgres: checkpointer
           ├─57183 postgres: background writer
           ├─57184 postgres: walwriter
           ├─57185 postgres: autovacuum launcher
           ├─57186 postgres: stats collector
           ├─57187 postgres: logical replication launcher
           ├─57241 postgres: pulp pulpcore 127.0.0.1(57298) idle
           ├─57243 postgres: pulp pulpcore 127.0.0.1(57314) idle
           ├─57248 postgres: pulp pulpcore 127.0.0.1(57330) idle
           ├─57253 postgres: pulp pulpcore 127.0.0.1(57334) idle
           ├─57268 postgres: pulp pulpcore 127.0.0.1(57348) idle
           ├─57307 postgres: pulp pulpcore 127.0.0.1(57364) idle
           ├─57342 postgres: pulp pulpcore 127.0.0.1(57376) idle
           ├─57357 postgres: pulp pulpcore 127.0.0.1(57388) idle
           ├─57369 postgres: pulp pulpcore 127.0.0.1(57396) idle
           ├─57373 postgres: pulp pulpcore 127.0.0.1(57404) idle
           ├─57375 postgres: pulp pulpcore 127.0.0.1(57408) idle
           ├─57383 postgres: pulp pulpcore 127.0.0.1(57412) idle
           ├─57390 postgres: pulp pulpcore 127.0.0.1(57422) idle
           ├─57391 postgres: pulp pulpcore 127.0.0.1(57430) idle
           ├─57393 postgres: pulp pulpcore 127.0.0.1(57434) idle
           ├─57396 postgres: pulp pulpcore 127.0.0.1(57446) idle
           ├─57398 postgres: pulp pulpcore 127.0.0.1(57448) idle
           ├─57586 postgres: pulp pulpcore 127.0.0.1(57528) idle
           ├─57588 postgres: pulp pulpcore 127.0.0.1(57530) idle
           ├─57589 postgres: pulp pulpcore 127.0.0.1(57534) idle
           ├─57590 postgres: pulp pulpcore 127.0.0.1(57550) idle
           ├─57591 postgres: pulp pulpcore 127.0.0.1(57560) idle
           ├─57592 postgres: pulp pulpcore 127.0.0.1(57562) idle
           ├─57593 postgres: pulp pulpcore 127.0.0.1(57566) idle
           ├─57594 postgres: pulp pulpcore 127.0.0.1(57576) idle
           ├─57605 postgres: candlepin candlepin 127.0.0.1(57592) idle in transaction
           ├─57632 postgres: candlepin candlepin 127.0.0.1(47760) idle
           ├─57633 postgres: candlepin candlepin 127.0.0.1(47758) idle
           ├─57634 postgres: candlepin candlepin 127.0.0.1(47756) idle
           ├─57635 postgres: candlepin candlepin 127.0.0.1(47788) idle
           ├─57636 postgres: candlepin candlepin 127.0.0.1(47786) idle
           ├─57684 postgres: foreman foreman [local] idle
           ├─57764 postgres: foreman foreman [local] idle
           ├─57771 postgres: foreman foreman [local] idle
           ├─57777 postgres: foreman foreman [local] idle
           ├─57785 postgres: foreman foreman [local] idle
           ├─57790 postgres: foreman foreman [local] idle
           ├─57797 postgres: foreman foreman [local] idle
           ├─57799 postgres: foreman foreman [local] idle
           ├─57800 postgres: foreman foreman [local] idle
           ├─57806 postgres: foreman foreman [local] idle
           ├─57807 postgres: foreman foreman [local] idle
           ├─57813 postgres: foreman foreman [local] idle
           ├─57814 postgres: foreman foreman [local] idle
           ├─57817 postgres: foreman foreman [local] idle
           ├─57821 postgres: foreman foreman [local] idle
           ├─57824 postgres: foreman foreman [local] idle
           ├─57825 postgres: foreman foreman [local] idle
           ├─57828 postgres: foreman foreman [local] idle
           ├─57832 postgres: foreman foreman [local] idle
           ├─57839 postgres: foreman foreman [local] idle
           ├─57840 postgres: foreman foreman [local] idle
           ├─57845 postgres: foreman foreman [local] idle
           ├─57846 postgres: foreman foreman [local] idle
           ├─57847 postgres: foreman foreman [local] idle
           ├─57854 postgres: foreman foreman [local] idle
           ├─57855 postgres: foreman foreman [local] idle
           ├─57857 postgres: foreman foreman [local] idle
           ├─57864 postgres: foreman foreman [local] idle
           ├─57867 postgres: foreman foreman [local] idle
           ├─57868 postgres: foreman foreman [local] idle
           ├─57869 postgres: foreman foreman [local] idle
           ├─57876 postgres: foreman foreman [local] idle
           ├─57877 postgres: foreman foreman [local] idle
           ├─57881 postgres: foreman foreman [local] idle
           ├─57891 postgres: foreman foreman [local] idle
           ├─57895 postgres: foreman foreman [local] idle
           ├─57902 postgres: foreman foreman [local] idle
           ├─57904 postgres: foreman foreman [local] idle
           ├─57907 postgres: foreman foreman [local] idle
           ├─57908 postgres: foreman foreman [local] idle
           ├─57929 postgres: foreman foreman [local] idle
           ├─57930 postgres: foreman foreman [local] idle
           ├─57931 postgres: foreman foreman [local] idle
           ├─57942 postgres: foreman foreman [local] idle
           ├─57945 postgres: foreman foreman [local] idle
           ├─57946 postgres: foreman foreman [local] idle
           ├─57956 postgres: foreman foreman [local] idle
           ├─57967 postgres: foreman foreman [local] idle
           ├─57969 postgres: foreman foreman [local] idle
           ├─57978 postgres: foreman foreman [local] idle
           ├─57980 postgres: foreman foreman [local] idle
           ├─57982 postgres: foreman foreman [local] idle
           ├─57986 postgres: foreman foreman [local] idle
           ├─57990 postgres: foreman foreman [local] idle
           ├─58011 postgres: foreman foreman [local] idle
           ├─58021 postgres: foreman foreman [local] idle
           ├─58025 postgres: foreman foreman [local] idle
           ├─58032 postgres: foreman foreman [local] idle
           ├─58035 postgres: foreman foreman [local] idle
           ├─58042 postgres: foreman foreman [local] idle
           ├─58045 postgres: foreman foreman [local] idle
           ├─58055 postgres: foreman foreman [local] idle
           ├─58059 postgres: foreman foreman [local] idle
           ├─58061 postgres: foreman foreman [local] idle
           ├─58063 postgres: foreman foreman [local] idle
           ├─58068 postgres: foreman foreman [local] idle
           ├─58069 postgres: foreman foreman [local] idle
           ├─58082 postgres: foreman foreman [local] idle
           ├─58094 postgres: foreman foreman [local] idle
           ├─58097 postgres: foreman foreman [local] idle
           ├─58122 postgres: foreman foreman [local] idle
           ├─58145 postgres: foreman foreman [local] idle
           ├─58148 postgres: foreman foreman [local] idle
           ├─58150 postgres: foreman foreman [local] idle
           ├─58151 postgres: foreman foreman [local] idle
           ├─58158 postgres: foreman foreman [local] idle
           ├─58161 postgres: foreman foreman [local] idle
           ├─58166 postgres: foreman foreman [local] idle
           ├─58170 postgres: foreman foreman [local] idle
           ├─58173 postgres: foreman foreman [local] idle
           ├─58178 postgres: foreman foreman [local] idle
           ├─58182 postgres: foreman foreman [local] idle
           ├─58190 postgres: foreman foreman [local] idle
           ├─58195 postgres: foreman foreman [local] idle
           ├─58196 postgres: foreman foreman [local] idle
           ├─58197 postgres: foreman foreman [local] idle
           ├─58201 postgres: foreman foreman [local] idle
           ├─58204 postgres: foreman foreman [local] idle
           ├─58212 postgres: foreman foreman [local] idle
           ├─58216 postgres: foreman foreman [local] idle
           ├─58218 postgres: foreman foreman [local] idle
           ├─58221 postgres: foreman foreman [local] idle
           ├─58224 postgres: foreman foreman [local] idle
           ├─58234 postgres: foreman foreman [local] idle
           ├─58238 postgres: foreman foreman [local] idle
           ├─58239 postgres: foreman foreman [local] idle
           ├─58243 postgres: foreman foreman [local] idle
           ├─58244 postgres: foreman foreman [local] idle
           ├─58246 postgres: foreman foreman [local] idle
           ├─58252 postgres: foreman foreman [local] idle
           ├─58255 postgres: foreman foreman [local] idle
           ├─58258 postgres: foreman foreman [local] idle
           ├─58261 postgres: foreman foreman [local] idle
           ├─58278 postgres: foreman foreman [local] idle
           ├─58290 postgres: foreman foreman [local] idle
           ├─58294 postgres: foreman foreman [local] idle
           ├─58299 postgres: foreman foreman [local] idle
           ├─58302 postgres: foreman foreman [local] idle
           ├─58321 postgres: foreman foreman [local] idle
           ├─58323 postgres: foreman foreman [local] idle
           ├─58345 postgres: foreman foreman [local] idle
           ├─58346 postgres: foreman foreman [local] idle
           ├─58353 postgres: foreman foreman [local] idle
           ├─58355 postgres: foreman foreman [local] idle
           ├─58360 postgres: foreman foreman [local] idle
           ├─58370 postgres: foreman foreman [local] idle
           ├─58372 postgres: foreman foreman [local] idle
           ├─58387 postgres: foreman foreman [local] idle
           ├─58389 postgres: foreman foreman [local] idle
           ├─58392 postgres: foreman foreman [local] idle
           ├─58397 postgres: foreman foreman [local] idle
           ├─58399 postgres: foreman foreman [local] idle
           ├─58414 postgres: foreman foreman [local] idle
           ├─58415 postgres: foreman foreman [local] idle
           ├─58418 postgres: foreman foreman [local] idle
           ├─58421 postgres: foreman foreman [local] idle
           ├─58429 postgres: foreman foreman [local] idle
           ├─58432 postgres: foreman foreman [local] idle
           ├─58443 postgres: foreman foreman [local] idle
           ├─58446 postgres: foreman foreman [local] idle
           ├─58454 postgres: foreman foreman [local] idle
           ├─58457 postgres: foreman foreman [local] idle
           ├─58461 postgres: foreman foreman [local] idle
           ├─58475 postgres: foreman foreman [local] idle
           ├─58479 postgres: foreman foreman [local] idle
           ├─58484 postgres: foreman foreman [local] idle
           ├─58485 postgres: foreman foreman [local] idle
           ├─58487 postgres: foreman foreman [local] idle
           ├─58502 postgres: foreman foreman [local] idle
           ├─58506 postgres: foreman foreman [local] idle
           ├─58507 postgres: foreman foreman [local] idle
           ├─58519 postgres: foreman foreman [local] idle
           ├─58521 postgres: foreman foreman [local] idle
           ├─58526 postgres: foreman foreman [local] idle
           ├─58542 postgres: foreman foreman [local] idle
           ├─58545 postgres: foreman foreman [local] idle
           ├─58547 postgres: foreman foreman [local] idle
           ├─58554 postgres: foreman foreman [local] idle
           ├─58555 postgres: foreman foreman [local] idle
           ├─58560 postgres: foreman foreman [local] idle
           ├─58562 postgres: foreman foreman [local] idle
           ├─58573 postgres: foreman foreman [local] idle
           ├─58575 postgres: foreman foreman [local] idle
           ├─58577 postgres: foreman foreman [local] idle
           ├─58578 postgres: foreman foreman [local] idle
           ├─58594 postgres: foreman foreman [local] idle
           ├─58595 postgres: foreman foreman [local] idle
           ├─58600 postgres: foreman foreman [local] idle
           ├─58603 postgres: foreman foreman [local] idle
           ├─58609 postgres: foreman foreman [local] idle
           ├─58612 postgres: foreman foreman [local] idle
           ├─58615 postgres: foreman foreman [local] idle
           ├─58624 postgres: foreman foreman [local] idle
           ├─58626 postgres: foreman foreman [local] idle
           ├─58627 postgres: foreman foreman [local] idle
           ├─58630 postgres: foreman foreman [local] idle
           ├─58638 postgres: foreman foreman [local] idle
           ├─58648 postgres: foreman foreman [local] idle
           ├─58650 postgres: foreman foreman [local] idle
           ├─58653 postgres: foreman foreman [local] idle
           ├─58658 postgres: foreman foreman [local] idle
           ├─58668 postgres: foreman foreman [local] idle
           ├─58669 postgres: foreman foreman [local] idle
           ├─58673 postgres: foreman foreman [local] idle
           ├─58676 postgres: foreman foreman [local] idle
           ├─58677 postgres: foreman foreman [local] idle
           ├─58684 postgres: foreman foreman [local] idle
           ├─58685 postgres: foreman foreman [local] idle
           ├─58687 postgres: foreman foreman [local] idle
           ├─58689 postgres: foreman foreman [local] idle
           ├─58693 postgres: foreman foreman [local] idle
           ├─58699 postgres: foreman foreman [local] idle
           ├─58701 postgres: foreman foreman [local] idle
           ├─58704 postgres: foreman foreman [local] idle
           ├─58707 postgres: foreman foreman [local] idle
           ├─58725 postgres: foreman foreman [local] idle
           ├─58730 postgres: foreman foreman [local] idle
           ├─58731 postgres: foreman foreman [local] idle
           ├─58732 postgres: foreman foreman [local] idle
           ├─58733 postgres: foreman foreman [local] idle
           ├─58734 postgres: foreman foreman [local] idle
           ├─58753 postgres: foreman foreman [local] idle
           ├─58754 postgres: foreman foreman [local] idle
           ├─58835 postgres: foreman foreman [local] idle
           ├─58839 postgres: foreman foreman [local] idle
           ├─58840 postgres: foreman foreman [local] idle
           ├─58841 postgres: foreman foreman [local] idle
           ├─58876 postgres: foreman foreman [local] idle
           ├─58880 postgres: foreman foreman [local] idle
           ├─58881 postgres: foreman foreman [local] idle
           ├─58894 postgres: foreman foreman [local] idle
           ├─58898 postgres: foreman foreman [local] idle
           └─58899 postgres: foreman foreman [local] idle

Dec 07 10:14:31 sjprdsatapp02.cadence.com systemd[1]: Starting PostgreSQL database server...
Dec 07 10:14:31 sjprdsatapp02.cadence.com postmaster[57179]: 2023-12-07 10:14:31 PST LOG:  starting PostgreSQL 12.9 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4), 64-bit
Dec 07 10:14:31 sjprdsatapp02.cadence.com postmaster[57179]: 2023-12-07 10:14:31 PST LOG:  listening on IPv4 address "127.0.0.1", port 5432
Dec 07 10:14:31 sjprdsatapp02.cadence.com postmaster[57179]: 2023-12-07 10:14:31 PST LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Dec 07 10:14:31 sjprdsatapp02.cadence.com postmaster[57179]: 2023-12-07 10:14:31 PST LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
Dec 07 10:14:31 sjprdsatapp02.cadence.com postmaster[57179]: 2023-12-07 10:14:31 PST LOG:  redirecting log output to logging collector process
Dec 07 10:14:31 sjprdsatapp02.cadence.com postmaster[57179]: 2023-12-07 10:14:31 PST HINT:  Future log output will appear in directory "log".
Dec 07 10:14:31 sjprdsatapp02.cadence.com systemd[1]: Started PostgreSQL database server.
/ displaying pulpcore-api
● pulpcore-api.service - Pulp API Server
   Loaded: loaded (/etc/systemd/system/pulpcore-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:35 PST; 1min 4s ago
 Main PID: 57208 (gunicorn)
   Status: "Gunicorn arbiter booted"
    Tasks: 6 (limit: 3355442)
   Memory: 149.9M
   CGroup: /system.slice/pulpcore-api.service
           ├─57208 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 800 --max-requests-jitter 100 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─57263 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 800 --max-requests-jitter 100 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─57335 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 800 --max-requests-jitter 100 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─57358 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 800 --max-requests-jitter 100 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─57360 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 800 --max-requests-jitter 100 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           └─57363 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 800 --max-requests-jitter 100 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"

Dec 07 10:14:32 sjprdsatapp02.cadence.com systemd[1]: Starting Pulp API Server...
Dec 07 10:14:35 sjprdsatapp02.cadence.com pulpcore-api[57208]: [2023-12-07 18:14:35 +0000] [57208] [INFO] Starting gunicorn 20.1.0
Dec 07 10:14:35 sjprdsatapp02.cadence.com pulpcore-api[57208]: [2023-12-07 18:14:35 +0000] [57208] [INFO] Listening at: unix:/run/pulpcore-api.sock (57208)
Dec 07 10:14:35 sjprdsatapp02.cadence.com pulpcore-api[57208]: [2023-12-07 18:14:35 +0000] [57208] [INFO] Using worker: sync
Dec 07 10:14:35 sjprdsatapp02.cadence.com systemd[1]: Started Pulp API Server.
Dec 07 10:14:35 sjprdsatapp02.cadence.com pulpcore-api[57263]: [2023-12-07 18:14:35 +0000] [57263] [INFO] Booting worker with pid: 57263
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-api[57335]: [2023-12-07 18:14:36 +0000] [57335] [INFO] Booting worker with pid: 57335
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-api[57358]: [2023-12-07 18:14:36 +0000] [57358] [INFO] Booting worker with pid: 57358
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-api[57360]: [2023-12-07 18:14:36 +0000] [57360] [INFO] Booting worker with pid: 57360
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-api[57363]: [2023-12-07 18:14:36 +0000] [57363] [INFO] Booting worker with pid: 57363
/ displaying pulpcore-content
● pulpcore-content.service - Pulp Content App
   Loaded: loaded (/etc/systemd/system/pulpcore-content.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:35 PST; 1min 4s ago
 Main PID: 57211 (gunicorn)
   Status: "Gunicorn arbiter booted"
    Tasks: 35 (limit: 3355442)
   Memory: 535.3M
   CGroup: /system.slice/pulpcore-content.service
           ├─57211 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57232 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57233 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57236 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57245 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57246 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57249 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57275 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57311 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57346 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57359 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57361 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57366 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57371 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57376 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57378 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─57380 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           └─57381 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -

Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57311]: [2023-12-07 18:14:36 +0000] [57311] [INFO] Booting worker with pid: 57311
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57346]: [2023-12-07 18:14:36 +0000] [57346] [INFO] Booting worker with pid: 57346
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57359]: [2023-12-07 18:14:36 +0000] [57359] [INFO] Booting worker with pid: 57359
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57361]: [2023-12-07 18:14:36 +0000] [57361] [INFO] Booting worker with pid: 57361
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57366]: [2023-12-07 18:14:36 +0000] [57366] [INFO] Booting worker with pid: 57366
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57371]: [2023-12-07 18:14:36 +0000] [57371] [INFO] Booting worker with pid: 57371
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57376]: [2023-12-07 18:14:36 +0000] [57376] [INFO] Booting worker with pid: 57376
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57378]: [2023-12-07 18:14:36 +0000] [57378] [INFO] Booting worker with pid: 57378
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57380]: [2023-12-07 18:14:36 +0000] [57380] [INFO] Booting worker with pid: 57380
Dec 07 10:14:36 sjprdsatapp02.cadence.com pulpcore-content[57381]: [2023-12-07 18:14:36 +0000] [57381] [INFO] Booting worker with pid: 57381
/ displaying pulpcore-worker@1.service
● pulpcore-worker@1.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:35 PST; 1min 4s ago
 Main PID: 57286 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.4M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
           └─57286 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:35 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-1[57286]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-1[57286]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57286@sjprdsatapp02.cadence.com' discovered
/ displaying pulpcore-worker@2.service
● pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:35 PST; 1min 4s ago
 Main PID: 57287 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.3M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
           └─57287 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:35 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-2[57287]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-2[57287]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57287@sjprdsatapp02.cadence.com' discovered
/ displaying pulpcore-worker@3.service
● pulpcore-worker@3.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:35 PST; 1min 4s ago
 Main PID: 57295 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.4M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@3.service
           └─57295 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:35 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-3[57295]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-3[57295]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57295@sjprdsatapp02.cadence.com' discovered
/ displaying pulpcore-worker@4.service
● pulpcore-worker@4.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:35 PST; 1min 4s ago
 Main PID: 57299 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.3M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
           └─57299 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:35 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-4[57299]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-4[57299]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57299@sjprdsatapp02.cadence.com' discovered
/ displaying pulpcore-worker@5.service
● pulpcore-worker@5.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:36 PST; 1min 4s ago
 Main PID: 57304 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.3M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@5.service
           └─57304 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:36 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-5[57304]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-5[57304]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57304@sjprdsatapp02.cadence.com' discovered
/ displaying pulpcore-worker@6.service
● pulpcore-worker@6.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:35 PST; 1min 4s ago
 Main PID: 57298 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.3M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@6.service
           └─57298 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:35 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-6[57298]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-6[57298]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57298@sjprdsatapp02.cadence.com' discovered
/ displaying pulpcore-worker@7.service
● pulpcore-worker@7.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:36 PST; 1min 4s ago
 Main PID: 57303 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.3M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@7.service
           └─57303 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:36 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-7[57303]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-7[57303]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57303@sjprdsatapp02.cadence.com' discovered
/ displaying pulpcore-worker@8.service
● pulpcore-worker@8.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:36 PST; 1min 4s ago
 Main PID: 57305 (pulpcore-worker)
    Tasks: 1 (limit: 3355442)
   Memory: 101.3M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@8.service
           └─57305 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Dec 07 10:14:36 sjprdsatapp02.cadence.com systemd[1]: Started Pulp Worker.
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-8[57305]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Dec 07 10:14:38 sjprdsatapp02.cadence.com pulpcore-worker-8[57305]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '57305@sjprdsatapp02.cadence.com' discovered
/ displaying tomcat
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:36 PST; 1min 4s ago
 Main PID: 57310 (java)
    Tasks: 70 (limit: 3355442)
   Memory: 1.0G
   CGroup: /system.slice/tomcat.service
           └─57310 /usr/lib/jvm/jre-17/bin/java -Xms1024m -Xmx4096m -Dcom.redhat.fips=false -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1137)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Dec 07 10:15:37 sjprdsatapp02.cadence.com server[57310]:                 at java.base/java.lang.Thread.run(Thread.java:833)
/ displaying dynflow-sidekiq@orchestrator
● dynflow-sidekiq@orchestrator.service - Foreman jobs daemon - orchestrator on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:51 PST; 48s ago
     Docs: https://theforeman.org
 Main PID: 57566 (sidekiq)
   Status: "Everything ready for world: 6115895e-ce08-407d-867f-85e5c011796d"
    Tasks: 13 (limit: 3355442)
   Memory: 367.7M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@orchestrator.service
           └─57566 sidekiq 6.3.1  [0 of 1 busy]

Dec 07 10:14:37 sjprdsatapp02.cadence.com systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Dec 07 10:14:37 sjprdsatapp02.cadence.com dynflow-sidekiq@orchestrator[57566]: 2023-12-07T18:14:37.879Z pid=57566 tid=1936 INFO: Enabling systemd notification integration
Dec 07 10:14:38 sjprdsatapp02.cadence.com dynflow-sidekiq@orchestrator[57566]: 2023-12-07T18:14:38.604Z pid=57566 tid=1936 INFO: Booting Sidekiq 6.3.1 with redis options {:url=>"redis://localhost:6379/0"}
Dec 07 10:14:38 sjprdsatapp02.cadence.com dynflow-sidekiq@orchestrator[57566]: 2023-12-07T18:14:38.605Z pid=57566 tid=1936 INFO: GitLab reliable fetch activated!
Dec 07 10:14:51 sjprdsatapp02.cadence.com systemd[1]: Started Foreman jobs daemon - orchestrator on sidekiq.
/ displaying foreman
● foreman.service - Foreman
   Loaded: loaded (/usr/lib/systemd/system/foreman.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/foreman.service.d
           └─installer.conf
   Active: active (running) since Thu 2023-12-07 10:14:50 PST; 49s ago
     Docs: https://theforeman.org
 Main PID: 57417 (rails)
   Status: "Puma 6.3.1: cluster: 48/48, worker_status: [{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog }]"
    Tasks: 918 (limit: 3355442)
   Memory: 3.7G
   CGroup: /system.slice/foreman.service
           ├─57417 puma 6.3.1 (unix:///run/foreman.sock) [foreman]
           ├─57701 puma: cluster worker 0: 57417 [foreman]
           ├─57705 puma: cluster worker 1: 57417 [foreman]
           ├─57710 puma: cluster worker 2: 57417 [foreman]
           ├─57716 puma: cluster worker 3: 57417 [foreman]
           ├─57721 puma: cluster worker 4: 57417 [foreman]
           ├─57727 puma: cluster worker 5: 57417 [foreman]
           ├─57733 puma: cluster worker 6: 57417 [foreman]
           ├─57739 puma: cluster worker 7: 57417 [foreman]
           ├─57745 puma: cluster worker 8: 57417 [foreman]
           ├─57752 puma: cluster worker 9: 57417 [foreman]
           ├─57758 puma: cluster worker 10: 57417 [foreman]
           ├─57765 puma: cluster worker 11: 57417 [foreman]
           ├─57773 puma: cluster worker 12: 57417 [foreman]
           ├─57780 puma: cluster worker 13: 57417 [foreman]
           ├─57792 puma: cluster worker 14: 57417 [foreman]
           ├─57804 puma: cluster worker 15: 57417 [foreman]
           ├─57820 puma: cluster worker 16: 57417 [foreman]
           ├─57833 puma: cluster worker 17: 57417 [foreman]
           ├─57850 puma: cluster worker 18: 57417 [foreman]
           ├─57863 puma: cluster worker 19: 57417 [foreman]
           ├─57882 puma: cluster worker 20: 57417 [foreman]
           ├─57903 puma: cluster worker 21: 57417 [foreman]
           ├─57928 puma: cluster worker 22: 57417 [foreman]
           ├─57952 puma: cluster worker 23: 57417 [foreman]
           ├─57979 puma: cluster worker 24: 57417 [foreman]
           ├─58016 puma: cluster worker 25: 57417 [foreman]
           ├─58040 puma: cluster worker 26: 57417 [foreman]
           ├─58065 puma: cluster worker 27: 57417 [foreman]
           ├─58080 puma: cluster worker 28: 57417 [foreman]
           ├─58096 puma: cluster worker 29: 57417 [foreman]
           ├─58126 puma: cluster worker 30: 57417 [foreman]
           ├─58157 puma: cluster worker 31: 57417 [foreman]
           ├─58185 puma: cluster worker 32: 57417 [foreman]
           ├─58210 puma: cluster worker 33: 57417 [foreman]
           ├─58233 puma: cluster worker 34: 57417 [foreman]
           ├─58248 puma: cluster worker 35: 57417 [foreman]
           ├─58259 puma: cluster worker 36: 57417 [foreman]
           ├─58291 puma: cluster worker 37: 57417 [foreman]
           ├─58331 puma: cluster worker 38: 57417 [foreman]
           ├─58358 puma: cluster worker 39: 57417 [foreman]
           ├─58385 puma: cluster worker 40: 57417 [foreman]
           ├─58402 puma: cluster worker 41: 57417 [foreman]
           ├─58424 puma: cluster worker 42: 57417 [foreman]
           ├─58450 puma: cluster worker 43: 57417 [foreman]
           ├─58471 puma: cluster worker 44: 57417 [foreman]
           ├─58491 puma: cluster worker 45: 57417 [foreman]
           ├─58518 puma: cluster worker 46: 57417 [foreman]
           └─58546 puma: cluster worker 47: 57417 [foreman]

Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 39 (PID: 58358) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 40 (PID: 58385) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 41 (PID: 58402) booted in 0.11s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 42 (PID: 58424) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 43 (PID: 58450) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 44 (PID: 58471) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 45 (PID: 58491) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 46 (PID: 58518) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com foreman[57417]: [57417] - Worker 47 (PID: 58546) booted in 0.1s, phase: 0
Dec 07 10:14:50 sjprdsatapp02.cadence.com systemd[1]: Started Foreman.
/ displaying httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:14:37 PST; 1min 3s ago
     Docs: man:httpd.service(8)
 Main PID: 57425 (httpd)
   Status: "Running, listening on: port 80, port 443"
    Tasks: 85 (limit: 3355442)
   Memory: 22.4M
   CGroup: /system.slice/httpd.service
           ├─57425 /usr/sbin/httpd -DFOREGROUND
           ├─57448 /usr/sbin/httpd -DFOREGROUND
           └─57449 /usr/sbin/httpd -DFOREGROUND

Dec 07 10:14:37 sjprdsatapp02.cadence.com systemd[1]: Starting The Apache HTTP Server...
Dec 07 10:14:37 sjprdsatapp02.cadence.com systemd[1]: Started The Apache HTTP Server.
Dec 07 10:14:37 sjprdsatapp02.cadence.com httpd[57425]: Server configured, listening on: port 80, port 443
/ displaying dynflow-sidekiq@worker-1
● dynflow-sidekiq@worker-1.service - Foreman jobs daemon - worker-1 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:15:06 PST; 34s ago
     Docs: https://theforeman.org
 Main PID: 58845 (sidekiq)
   Status: "Everything ready for world: 7a612d08-d91f-40cb-9d6f-dbb94b5e819b"
    Tasks: 16 (limit: 3355442)
   Memory: 364.3M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-1.service
           └─58845 sidekiq 6.3.1  [0 of 5 busy]

Dec 07 10:14:52 sjprdsatapp02.cadence.com systemd[1]: Starting Foreman jobs daemon - worker-1 on sidekiq...
Dec 07 10:14:52 sjprdsatapp02.cadence.com dynflow-sidekiq@worker-1[58845]: 2023-12-07T18:14:52.513Z pid=58845 tid=18ht INFO: Enabling systemd notification integration
Dec 07 10:14:53 sjprdsatapp02.cadence.com dynflow-sidekiq@worker-1[58845]: 2023-12-07T18:14:53.172Z pid=58845 tid=18ht INFO: Booting Sidekiq 6.3.1 with redis options {:url=>"redis://localhost:6379/0"}
Dec 07 10:14:53 sjprdsatapp02.cadence.com dynflow-sidekiq@worker-1[58845]: 2023-12-07T18:14:53.172Z pid=58845 tid=18ht INFO: GitLab reliable fetch activated!
Dec 07 10:15:06 sjprdsatapp02.cadence.com systemd[1]: Started Foreman jobs daemon - worker-1 on sidekiq.
/ displaying dynflow-sidekiq@worker-hosts-queue-1
● dynflow-sidekiq@worker-hosts-queue-1.service - Foreman jobs daemon - worker-hosts-queue-1 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:15:06 PST; 33s ago
     Docs: https://theforeman.org
 Main PID: 58852 (sidekiq)
   Status: "Everything ready for world: b4dcbe8f-aa0c-4870-ad3d-203a931d980c"
    Tasks: 16 (limit: 3355442)
   Memory: 362.9M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-hosts-queue-1.service
           └─58852 sidekiq 6.3.1  [0 of 5 busy]

Dec 07 10:14:53 sjprdsatapp02.cadence.com systemd[1]: Starting Foreman jobs daemon - worker-hosts-queue-1 on sidekiq...
Dec 07 10:14:53 sjprdsatapp02.cadence.com dynflow-sidekiq@worker-hosts-queue-1[58852]: 2023-12-07T18:14:53.535Z pid=58852 tid=18jc INFO: Enabling systemd notification integration
Dec 07 10:14:54 sjprdsatapp02.cadence.com dynflow-sidekiq@worker-hosts-queue-1[58852]: 2023-12-07T18:14:54.199Z pid=58852 tid=18jc INFO: Booting Sidekiq 6.3.1 with redis options {:url=>"redis://localhost:6379/0"}
Dec 07 10:14:54 sjprdsatapp02.cadence.com dynflow-sidekiq@worker-hosts-queue-1[58852]: 2023-12-07T18:14:54.200Z pid=58852 tid=18jc INFO: GitLab reliable fetch activated!
Dec 07 10:15:06 sjprdsatapp02.cadence.com systemd[1]: Started Foreman jobs daemon - worker-hosts-queue-1 on sidekiq.
/ displaying foreman-proxy
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-12-07 10:15:07 PST; 33s ago
 Main PID: 58911 (smart-proxy)
    Tasks: 2 (limit: 3355442)
   Memory: 38.7M
   CGroup: /system.slice/foreman-proxy.service
           └─58911 /usr/bin/ruby /usr/share/foreman-proxy/bin/smart-proxy

Dec 07 10:15:06 sjprdsatapp02.cadence.com systemd[1]: Starting Foreman Proxy...
Dec 07 10:15:07 sjprdsatapp02.cadence.com systemd[1]: Started Foreman Proxy.
/ All services are running                                            [OK]
--------------------------------------------------------------------------------

hammer command still shows same error.

[root@sjprdsatapp02 ~]# hammer status
Version:           3.8.0
API Version:       v2
Database:
    Status:          ok
    Server Response: Duration: 0ms
Plugins:
 1) Name:    foreman-tasks
    Version: 8.3.3
 2) Name:    foreman_remote_execution
    Version: 11.1.1
 3) Name:    katello
    Version: 4.10.0
Smart Proxies:
 1) Name:     sjprdsatapp02.cadence.com
    Version:  3.8.0
    Status:   ok
    Features:
     1) Name:    pulpcore
        Version: 3.2.0
     2) Name:    logs
        Version: 3.8.0
Compute Resources:

candlepin:
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:
    Status:          FAIL
    Server Response: Message: Katello::Errors::CandlepinNotRunning
candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:
    Status:          ok
    Server Response: Duration: 55ms
pulp3_content:
    Status:          ok
    Server Response: Duration: 55ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 3ms

sarang · December 8, 2023, 3:29am

Here is the foreman rake output when I log in to webui.

[root@sjprdsatapp02 ~]# foreman-rake errors:fetch_log request_id=599d2ea4
Foreman version: 3.8.0
Plugins:
 - foreman-tasks 8.3.3
 - foreman_remote_execution 11.1.1
 - katello 4.10.0

2023-12-07T19:23:55 [I|app|599d2ea4] Started GET "/" for 172.27.74.5 at 2023-12-07 19:23:55 -0800
2023-12-07T19:23:55 [W|app|599d2ea4] 404 Not Found
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
 599d2ea4 | /usr/share/ruby/net/http.rb:933:in `start'
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
 599d2ea4 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/ping.rb:264:in `backend_status'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/ping.rb:87:in `block in ping_candlepin_without_auth'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/ping.rb:136:in `exception_watch'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/ping.rb:86:in `ping_candlepin_without_auth'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/ping.rb:236:in `ping_services_for_capsule'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/ping.rb:24:in `ping!'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/services/katello/organization_creator.rb:20:in `create_all_organizations!'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/lib/katello/middleware/organization_created_enforcer.rb:12:in `call'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/lib/katello/middleware/event_daemon.rb:10:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 599d2ea4 | /usr/share/gems/gems/apipie-rails-1.2.2/lib/apipie/static_dispatcher.rb:68:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 599d2ea4 | /usr/share/gems/gems/apipie-rails-1.2.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/conditional_get.rb:27:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:37:in `call_app'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:28:in `call'
 599d2ea4 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 599d2ea4 | /usr/share/gems/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/lib/katello/prevent_json_parsing.rb:12:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 599d2ea4 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `public_send'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `method_missing'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/configuration.rb:270:in `call'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/request.rb:100:in `block in handle_request'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/thread_pool.rb:344:in `with_force_shutdown'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/request.rb:99:in `handle_request'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/server.rb:443:in `process_client'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/server.rb:245:in `block in run'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/thread_pool.rb:151:in `block in spawn_thread'
 599d2ea4 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-12-07T19:23:55 [I|app|599d2ea4] Processing by DashboardController#index as HTML
2023-12-07T19:23:55 [I|app|599d2ea4]   Rendered dashboard/index.html.erb within layouts/application (Duration: 31.7ms | Allocations: 6018)
2023-12-07T19:23:55 [I|app|599d2ea4]   Rendered layout layouts/application.html.erb (Duration: 31.8ms | Allocations: 6073)
2023-12-07T19:23:55 [W|app|599d2ea4] Katello::Errors::CandlepinNotRunning
2023-12-07T19:23:55 [I|app|599d2ea4] Backtrace for 'Katello::Errors::CandlepinNotRunning' error (ActionView::Template::Error): Katello::Errors::CandlepinNotRunning
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/lib/katello/resources/candlepin.rb:27:in `raise_rest_client_exception'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/lib/katello/http_resource.rb:94:in `rescue in issue_request'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/lib/katello/http_resource.rb:84:in `issue_request'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/lib/katello/http_resource.rb:47:in `block (2 levels) in singleton class'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/lib/katello/resources/candlepin/owner.rb:38:in `find'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/glue/candlepin/owner.rb:20:in `find_owner'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/glue/candlepin/owner.rb:31:in `owner_details'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/glue/candlepin/owner.rb:45:in `content_access_mode'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/glue/candlepin/owner.rb:54:in `block in simple_content_access?'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache.rb:752:in `block in save_block_result_to_cache'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache.rb:726:in `block in instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache.rb:726:in `instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache.rb:751:in `save_block_result_to_cache'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache.rb:343:in `fetch'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/models/katello/glue/candlepin/owner.rb:53:in `simple_content_access?'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/app/helpers/katello/concerns/dashboard_helper_extensions.rb:35:in `removed_widgets'
 599d2ea4 | /usr/share/foreman/app/helpers/dashboard_helper.rb:32:in `widgets_to_add'
 599d2ea4 | /usr/share/foreman/app/helpers/dashboard_helper.rb:16:in `block in dashboard_actions'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:45:in `block in capture'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:209:in `with_output_buffer'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:45:in `capture'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/helpers/tag_helper.rb:342:in `content_tag'
 599d2ea4 | /usr/share/foreman/app/helpers/dashboard_helper.rb:15:in `dashboard_actions'
 599d2ea4 | /usr/share/foreman/app/views/dashboard/index.html.erb:2:in `_f04f00c0ab1b1e49f0246aa59decddb2'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/base.rb:247:in `public_send'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/base.rb:247:in `_run'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/template.rb:154:in `block in render'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/template.rb:345:in `instrument_render_template'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/template.rb:152:in `render'
 599d2ea4 | /usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:61:in `block (2 levels) in render_template'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:56:in `block in render_template'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:71:in `block in render_with_layout'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:70:in `render_with_layout'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:55:in `render_template'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:11:in `render'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:117:in `block in _render_template'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/base.rb:273:in `in_rendering_context'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:116:in `_render_template'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/streaming.rb:218:in `_render_template'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:103:in `render_to_body'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/rendering.rb:25:in `render'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:36:in `render'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
 599d2ea4 | /usr/share/ruby/benchmark.rb:308:in `realtime'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:86:in `cleanup_view_runtime'
 599d2ea4 | /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:45:in `render'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 599d2ea4 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 599d2ea4 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 599d2ea4 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 599d2ea4 | /usr/share/gems/gems/audited-5.3.3/lib/audited/sweeper.rb:16:in `around'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 599d2ea4 | /usr/share/gems/gems/audited-5.3.3/lib/audited/sweeper.rb:16:in `around'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:137:in `run_callbacks'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 599d2ea4 | /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
 599d2ea4 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/lib/katello/middleware/event_daemon.rb:10:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 599d2ea4 | /usr/share/gems/gems/apipie-rails-1.2.2/lib/apipie/static_dispatcher.rb:68:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 599d2ea4 | /usr/share/gems/gems/apipie-rails-1.2.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/conditional_get.rb:27:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:37:in `call_app'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:28:in `call'
 599d2ea4 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 599d2ea4 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 599d2ea4 | /usr/share/gems/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 599d2ea4 | /usr/share/gems/gems/katello-4.10.0/lib/katello/prevent_json_parsing.rb:12:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
 599d2ea4 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 599d2ea4 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 599d2ea4 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `public_send'
 599d2ea4 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `method_missing'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
 599d2ea4 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/configuration.rb:270:in `call'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/request.rb:100:in `block in handle_request'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/thread_pool.rb:344:in `with_force_shutdown'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/request.rb:99:in `handle_request'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/server.rb:443:in `process_client'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/server.rb:245:in `block in run'
 599d2ea4 | /usr/share/gems/gems/puma-6.3.1/lib/puma/thread_pool.rb:151:in `block in spawn_thread'
 599d2ea4 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-12-07T19:23:55 [I|app|599d2ea4]   Rendered common/500.html.erb within layouts/application (Duration: 1.9ms | Allocations: 1092)
2023-12-07T19:23:55 [I|app|599d2ea4]   Rendered layouts/base.html.erb (Duration: 5.9ms | Allocations: 7206)
2023-12-07T19:23:55 [I|app|599d2ea4]   Rendered layout layouts/application.html.erb (Duration: 9.4ms | Allocations: 9259)
2023-12-07T19:23:55 [I|app|599d2ea4] Completed 500 Internal Server Error in 54ms (Views: 10.1ms | ActiveRecord: 3.4ms | Allocations: 20246)

The error i see is
2023-12-07T19:23:55 [W|app|599d2ea4] Katello::Errors::CandlepinNotRunning
2023-12-07T19:23:55 [I|app|599d2ea4] Backtrace for ‘Katello::Errors::CandlepinNotRunning’ error (ActionView::Template::Error): Katello::Errors::CandlepinNotRunning

its all in one installation. I do see candlepin db created during installation. Also see candlepin in ps.
However, not sure which service has failed and why this error ‘CandlepinNotRunning’

[root@sjprdsatapp02 ~]# ps -ef | grep -i candle
root       24869       1  0 00:47 ?        00:00:00 gpg-agent --homedir /var/cache/PackageKit/8.6/metadata/katello-candlepin-8-x86_64.tmp/gpgdir --use-standard-socket --daemon
postgres   79413   78987  0 19:23 ?        00:00:00 postgres: candlepin candlepin 127.0.0.1(35524) idle in transaction
postgres   79444   78987  0 19:23 ?        00:00:00 postgres: candlepin candlepin 127.0.0.1(35538) idle
postgres   79445   78987  0 19:23 ?        00:00:00 postgres: candlepin candlepin 127.0.0.1(35536) idle
postgres   79446   78987  0 19:23 ?        00:00:00 postgres: candlepin candlepin 127.0.0.1(35534) idle
postgres   79447   78987  0 19:23 ?        00:00:00 postgres: candlepin candlepin 127.0.0.1(35570) idle
postgres   79448   78987  0 19:23 ?        00:00:00 postgres: candlepin candlepin 127.0.0.1(35568) idle
postgres   80903   78987  0 19:27 ?        00:00:00 postgres: postgres candlepin [local] idle


[root@sjprdsatapp02 ~]# su - postgres
Last login: Thu Dec  7 03:05:56 PST 2023 on pts/1
[postgres@sjprdsatapp02 ~]$ psql
psql (12.9)
Type "help" for help.

postgres=# /l
postgres-# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+------------------------
 candlepin | postgres | UTF8     | en_US.utf8  | en_US.utf8  | =T/postgres           +
           |          |          |             |             | postgres=CTc/postgres +
           |          |          |             |             | candlepin=CTc/postgres
 foreman   | foreman  | UTF8     | en_US.utf8  | en_US.utf8  | =T/foreman            +
           |          |          |             |             | foreman=CTc/foreman
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 pulpcore  | postgres | UTF8     | en_US.utf8  | en_US.utf8  | =T/postgres           +
           |          |          |             |             | postgres=CTc/postgres +
           |          |          |             |             | pulp=CTc/postgres
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
(6 rows)

sarang · December 8, 2023, 6:14am

@jeremylenz Another interesting thing is ‘/var/log/candlepin/’ is empty. No log files inside.

sarang · December 8, 2023, 9:35am

@jeremylenz
Another interesting thing is ‘/var/log/candlepin/’ is empty. No log files inside.
—> This can be ignored. It occured as permissions of dir ‘var/log/candlepin’ were root:root.
Changed it to tomcat:tomcat. Now getting logs however, main issue is still there.