Upgrade from 3.10.0/4.12.1 to 3.11.1/4.13.1 makes foreman unusable

vchepkov · July 27, 2024, 12:37pm

I upgraded from 3.11.0/4.13.0 to 3.11.1/4.13.1 and have the same issue.
Installer stuck on Executing: foreman-rake upgrade:run

vchepkov · July 27, 2024, 2:26pm

It eventually finished, took more then two hours though

gvde · July 27, 2024, 3:10pm

How is the performance of API calls, e.g. https://foreman8.example.com/api/hosts/1 (replace the 1 with an existing id if you don’t have an host with id anymore)?

vchepkov · July 27, 2024, 3:43pm

You don’t need to measure api, clearly they are hosed. Even login takes minutes,

vchepkov · July 27, 2024, 3:45pm

There is message " Some services are not properly started. See the About page for more information", but About page won’t open, but foreman-maintain service status thinks everything is running

vchepkov · July 27, 2024, 3:57pm

# time hammer host list
---|---------------------|------------------|------------|--------------|-------------------|---------------|--------------|----------------------
ID | NAME                | OPERATING SYSTEM | HOST GROUP | IP           | MAC               | GLOBAL STATUS | CONTENT VIEW | LIFECYCLE ENVIRONMENT
---|---------------------|------------------|------------|--------------|-------------------|---------------|--------------|----------------------
1  | foreman.chepkov.lan | RHEL 9.4         |            | 192.168.1.77 | 52:54:00:38:4f:7d | OK            |              |                      
---|---------------------|------------------|------------|--------------|-------------------|---------------|--------------|----------------------

real	4m4.045s
user	0m1.094s
sys	0m0.177s

# time curl -I -k https://foreman.chepkov.lan/api/hosts/1
HTTP/2 401 
content-type: application/json; charset=utf-8
apipie-checksum: 77bd25b3dcff3ed026876881eb147cc1cdbea0b0
cache-control: no-cache
x-request-id: 0d48c1f9-2463-4bbb-ac14-d42ea2590178
x-runtime: 120.154458
strict-transport-security: max-age=631139040; includeSubdomains
x-frame-options: sameorigin
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-download-options: noopen
x-permitted-cross-domain-policies: none
content-security-policy: default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data:; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'
via: 2.0 foreman.chepkov.lan
date: Sat, 27 Jul 2024 15:54:33 GMT
server: Apache


real	2m0.249s
user	0m0.054s
sys	0m0.014s

evgeni · July 27, 2024, 6:21pm

Can you get us the output (and rough timing – “slow”/“quick”) for the following:

hammer ping
foreman-puma-status
pulp status
curl -k https://localhost:23443/candlepin/status
netstat -tulpen |grep java

vchepkov · July 28, 2024, 12:10pm

FYI, rebooting the server fixed the problem for me, but I have repeated the same procedure on another server and it is in the same state too. I can keep it like that for the diagnostic

# hammer ping
Error: Timed out reading data from server

# foreman-puma-status
317574 (/usr/share/foreman/tmp/puma.state) Version: 6.4.2/ruby2.7.8p225 | Uptime: 23h44m | Phase: 0 | Load: 1[█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]30 | Req: 222
 └ 317681 CPU:   0.0% Mem:  420 MB Uptime: 23h44m | Load: 0[░░░░░]5 | Req: 78
 └ 317684 CPU:   0.0% Mem:  413 MB Uptime: 23h44m | Load: 1[█░░░░]5 | Req: 36
 └ 317688 CPU:   0.0% Mem:  412 MB Uptime: 23h44m | Load: 0[░░░░░]5 | Req: 14
 └ 317691 CPU:   0.0% Mem:  418 MB Uptime: 23h44m | Load: 0[░░░░░]5 | Req: 37
 └ 317695 CPU:   0.0% Mem:  415 MB Uptime: 23h44m | Load: 0[░░░░░]5 | Req: 42
 └ 317698 CPU:   0.0% Mem:  412 MB Uptime: 23h44m | Load: 0[░░░░░]5 | Req: 15

# curl -k https://localhost:23443/candlepin/status
curl: (7) Failed to connect to localhost port 23443: Connection timed out

# netstat -tulpen |grep java
tcp6     101      0 127.0.0.1:23443         :::*                    LISTEN      91         5842385    317187/java

pulp status is quite verbose, I will upload it as a file

vchepkov · July 28, 2024, 12:16pm

Didn’t find attach file option, posted it here

gist.github.com

https://gist.github.com/vchepkov/c74674c24810433493042b7444b41ef5

gistfile1.txt

{"versions": [{"component": "core", "version": "3.49.10", "package": "pulpcore", "module": "pulpcore.app", "domain_compatible": true}, {"component": "certguard", "version": "3.49.10", "package": "pulpcore", "module": "pulp_certguard.app", "domain_compatible": true}, {"component": "file", "version": "3.49.10", "package": "pulpcore", "module": "pulp_file.app", "domain_compatible": true}, {"component": "rpm", "version": "3.26.1", "package": "pulp-rpm", "module": "pulp_rpm.app", "domain_compatible": true}, {"component": "python", "version": "3.11.1", "package": "pulp-python", "module": "pulp_python.app", "domain_compatible": false}, {"component": "deb", "version": "3.2.0", "package": "pulp_deb", "module": "pulp_deb.app", "domain_compatible": false}, {"component": "container", "version": "2.20.1", "package": "pulp-container", "module": "pulp_container.app", "domain_compatible": false}, {"component": "ansible", "version": "0.21.6", "package": "pulp-ansible", "module": "pulp_ansible.app", "domain_compatible": false}], "online_workers": [{"pulp_href": "/pulp/api/v3/workers/0190f426-4ae7-7eaf-8f76-f8df426bb017/", "pulp_created": "2024-07-27T12:24:28.666031Z", "pulp_last_updated": "2024-07-27T12:24:28.666049Z", "name": "318001@foreman.example.com", "last_heartbeat": "2024-07-28T12:11:01.847346Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}, "current_task": null}, {"pulp_href": "/pulp/api/v3/workers/0190f426-4b19-7552-b822-6f62de378cd7/", "pulp_created": "2024-07-27T12:24:28.698430Z", "pulp_last_updated": "2024-07-27T12:24:28.698446Z", "name": "317978@foreman.example.com", "last_heartbeat": "2024-07-28T12:11:01.837107Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}, "current_task": null}, {"pulp_href": "/pulp/api/v3/workers/0190f426-4c6e-7719-962e-cfae81e4551c/", "pulp_created": "2024-07-27T12:24:29.039517Z", "pulp_last_updated": "2024-07-27T12:24:29.039533Z", "name": "318023@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:54.318533Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}, "current_task": null}, {"pulp_href": "/pulp/api/v3/workers/0190f426-4d9f-7772-aa99-41bf99bed427/", "pulp_created": "2024-07-27T12:24:29.343973Z", "pulp_last_updated": "2024-07-27T12:24:29.343988Z", "name": "318045@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:54.304316Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}, "current_task": null}], "online_api_apps": [{"name": "318124@foreman.example.com", "last_heartbeat": "2024-07-28T12:11:03.200175Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318120@foreman.example.com", "last_heartbeat": "2024-07-28T12:11:03.201906Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318145@foreman.example.com", "last_heartbeat": "2024-07-28T12:11:03.198785Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318148@foreman.example.com", "last_heartbeat": "2024-07-28T12:11:03.198302Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318126@foreman.example.com", "last_heartbeat": "2024-07-28T12:11:03.195167Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}], "online_content_apps": [{"name": "318182@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:59.327055Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318222@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:56.939382Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318188@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:56.937167Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318225@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:56.938610Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318226@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:56.940175Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318208@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:56.940217Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318211@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:59.289906Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318214@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:59.332182Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}, {"name": "318219@foreman.example.com", "last_heartbeat": "2024-07-28T12:10:56.940852Z", "versions": {"deb": "3.2.0", "rpm": "3.26.1", "core": "3.49.10", "file": "3.49.10", "python": "3.11.1", "ansible": "0.21.6", "certguard": "3.49.10", "container": "2.20.1"}}], "database_connection": {"connected": true}, "redis_connection": {"connected": true}, "storage": {"total": 85857402880, "used": 1685782528, "free": 84171620352}, "content_settings": {"content_origin": "https://foreman.example.com", "content_path_prefix": "/pulp/content/"}, "domain_enabled": false}

gvde · July 28, 2024, 2:49pm

I am currently testing again and collecting logs.

I have just checked the database activity and noticed candlepin is stuck idle in transaction.

postgres=# select *  from pg_stat_activity where state <> 'idle';
 datid |  datname  |  pid  | leader_pid | usesysid |  usename  |    application_name    | client_addr | client_hostname | client_port |         backend_start         |          xact_start           |          query_start          |         state_change   
       | wait_event_type | wait_event |        state        | backend_xid | backend_xmin |                              query                              |  backend_type  
-------+-----------+-------+------------+----------+-----------+------------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+------------------------
-------+-----------------+------------+---------------------+-------------+--------------+-----------------------------------------------------------------+----------------
 16401 | candlepin | 24517 |            |    16384 | candlepin | PostgreSQL JDBC Driver | 127.0.0.1   |                 |       35824 | 2024-07-28 16:35:25.806063+02 | 2024-07-28 16:35:25.978385+02 | 2024-07-28 16:35:25.985594+02 | 2024-07-28 16:35:25.986
474+02 | Client          | ClientRead | idle in transaction |             |    550265702 | select setting from pg_settings where name = 'edb_redwood_date' | client backend
 19028 | postgres  | 26870 |            |       10 | postgres  | psql                   |             |                 |          -1 | 2024-07-28 16:44:10.56482+02  | 2024-07-28 16:46:57.800293+02 | 2024-07-28 16:46:57.800293+02 | 2024-07-28 16:46:57.800
295+02 |                 |            | active              |             |    550283473 | select *  from pg_stat_activity where state <> 'idle';          | client backend
(2 rows)

gvde · July 28, 2024, 3:09pm

O.K. I think it’s the same issue as in Feedback for Foreman 3.11 & Katello 4.13

I have updated, ran foreman-installer, stopped everything again, ran foreman-installer again, rebooted. No change.

Then I downgraded candlepin to 4.4.10 and restart tomcat.service and all of a sudden everything is back to normal, fast and smooth. No more timeouts. Everything is recovering.

Thus, it seems to me as if candlepin got stuck during startup, slowing everything else down, too.

I have got some logs collected, if needed, but revert the vm for now.

iballou · July 29, 2024, 5:41pm

Here’s a bugzilla on the Candlepin side to watch: 2300323 – candlepin-4.4.12-1 DB upgrade gets stuck

nikos_moum · July 31, 2024, 4:07pm

candlepin-4.4.13-1 is now available, which has the fix for the candlepin startup issue

vchepkov · July 31, 2024, 6:16pm

I don’t think it’s the case. I updated from 4.4.10 to 4.4.13 and watching

Executing: foreman-rake upgrade:run

for awhile now

lumarel · July 31, 2024, 6:40pm

The story is not yet fully over, lets give them a bit more time

dobrie2_gmu_edu · August 1, 2024, 7:19pm

Also having the same issue. I’ve just completed a series of upgrades (incrementally, from 3.6.1 → 3.6.2, and then through the rest of the major releases).

Foreman 3.9/Katello 4.11 was the last upgrade where the dashboard worked. In 3.10+4.12 I can log in, and use the navigation, but none of the screens ever come up. There didn’t seem to be any errors while foreman-install was running.

There does seem to be some kind of timeout issue. pulpcore-content, pulpcore-api and foreman-proxy time out when starting, but it looks like systemd tries (successfully) to restart pulpcore-content and pulpcore-api. I can start foreman-proxy once the system is up and Foreman has finished starting up everything else.

What information can I provide that’s useful? I’m probably going to revert to the snapshot I took of Foreman 3.9+Katello 4.11 and sit there until this is worked out.

Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: foreman-proxy.service: start operation timed out. Terminating.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: start operation timed out. Terminating.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: Failed with result 'timeout'.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: Failed to start Pulp Content App.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: start operation timed out. Terminating.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: foreman-proxy.service: Failed with result 'timeout'.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: Failed to start Foreman Proxy.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: Failed with result 'timeout'.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: Failed to start Pulp API Server.
Aug 01 14:59:18 themis.cecnet.gmu.edu systemd[1]: Starting SSSD Kerberos Cache Manager...
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: Service RestartSec=3s expired, scheduling restart.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: Scheduled restart job, restart counter is at 1.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: Service RestartSec=3s expired, scheduling restart.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: Scheduled restart job, restart counter is at 1.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Stopped Pulp API Server.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Starting Pulp API Server...
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Stopped Pulp Content App.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Starting Pulp Content App...
Aug 01 14:59:21 themis.cecnet.gmu.edu systemd[1]: Started SSSD Kerberos Cache Manager.
Aug 01 14:59:21 themis.cecnet.gmu.edu sssd_kcm[1964]: Starting up
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-1[1315]: 2024-08-01T18:59:42.068Z pid=1315 tid=cf INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-hosts-queue-1[1313]: 2024-08-01T18:59:42.070Z pid=1313 tid=cd INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@orchestrator[1311]: 2024-08-01T18:59:42.068Z pid=1311 tid=ar INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-hosts-queue-1[1313]: 2024-08-01T18:59:42.073Z pid=1313 tid=cd INFO: GitLab reliable fetch activated!
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@orchestrator[1311]: 2024-08-01T18:59:42.073Z pid=1311 tid=ar INFO: GitLab reliable fetch activated!
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-1[1315]: 2024-08-01T18:59:42.073Z pid=1315 tid=cf INFO: GitLab reliable fetch activated!
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-2[1218]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-3[1214]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-1[1213]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-4[1216]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker

dobrie2_gmu_edu · August 1, 2024, 7:30pm

ACTUALLY…

It looks like pulpcore-api is not working correctly:

Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4124]: [2024-08-01 19:27:59 +0000] [4124] [INFO] Starting gunicorn 20.1.0
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4124]: [2024-08-01 19:27:59 +0000] [4124] [INFO] Listening at: unix:/run/pulpcore-api.sock (4124)
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4124]: [2024-08-01 19:27:59 +0000] [4124] [INFO] Using worker: pulpcore.app.entrypoint.PulpApiWorker
Aug 01 15:27:59 themis.cecnet.gmu.edu systemd[1]: Started Pulp API Server.
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4136]: [2024-08-01 19:27:59 +0000] [4136] [INFO] Booting worker with pid: 4136
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4137]: [2024-08-01 19:27:59 +0000] [4137] [INFO] Booting worker with pid: 4137
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4138]: [2024-08-01 19:27:59 +0000] [4138] [INFO] Booting worker with pid: 4138
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4139]: [2024-08-01 19:27:59 +0000] [4139] [INFO] Booting worker with pid: 4139
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4140]: [2024-08-01 19:27:59 +0000] [4140] [INFO] Booting worker with pid: 4140
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4140]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4140@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4136]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4136@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4137]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4137@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4138]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4138@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4139]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4139@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.

gvde · August 1, 2024, 7:52pm

That is a different issue. Please open a separate thread or search if another thread covers it already. This thread is about an issue upgrading from 3.10 to 3.11.

dobrie2_gmu_edu · August 1, 2024, 8:06pm

Or, I could feel unwelcome and just go back to lurking.

lumarel · August 1, 2024, 8:12pm

You are not un-welcome, it just really looks like a different issue, to not clutter the thread it would be the best to move this to it’s own thread
(and if it’s in the end the same issue, finding out also doesn’t clutter the main thread)