Upgrade from 3.10.0/4.12.1 to 3.11.1/4.13.1 makes foreman unusable

Support
21

Also having the same issue. I’ve just completed a series of upgrades (incrementally, from 3.6.1 → 3.6.2, and then through the rest of the major releases).

Foreman 3.9/Katello 4.11 was the last upgrade where the dashboard worked. In 3.10+4.12 I can log in, and use the navigation, but none of the screens ever come up. There didn’t seem to be any errors while foreman-install was running.

There does seem to be some kind of timeout issue. pulpcore-content, pulpcore-api and foreman-proxy time out when starting, but it looks like systemd tries (successfully) to restart pulpcore-content and pulpcore-api. I can start foreman-proxy once the system is up and Foreman has finished starting up everything else.

What information can I provide that’s useful? I’m probably going to revert to the snapshot I took of Foreman 3.9+Katello 4.11 and sit there until this is worked out.

Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: foreman-proxy.service: start operation timed out. Terminating.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: start operation timed out. Terminating.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: Failed with result 'timeout'.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: Failed to start Pulp Content App.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: start operation timed out. Terminating.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: foreman-proxy.service: Failed with result 'timeout'.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: Failed to start Foreman Proxy.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: Failed with result 'timeout'.
Aug 01 14:59:17 themis.cecnet.gmu.edu systemd[1]: Failed to start Pulp API Server.
Aug 01 14:59:18 themis.cecnet.gmu.edu systemd[1]: Starting SSSD Kerberos Cache Manager...
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: Service RestartSec=3s expired, scheduling restart.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-content.service: Scheduled restart job, restart counter is at 1.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: Service RestartSec=3s expired, scheduling restart.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: pulpcore-api.service: Scheduled restart job, restart counter is at 1.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Stopped Pulp API Server.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Starting Pulp API Server...
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Stopped Pulp Content App.
Aug 01 14:59:20 themis.cecnet.gmu.edu systemd[1]: Starting Pulp Content App...
Aug 01 14:59:21 themis.cecnet.gmu.edu systemd[1]: Started SSSD Kerberos Cache Manager.
Aug 01 14:59:21 themis.cecnet.gmu.edu sssd_kcm[1964]: Starting up
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-1[1315]: 2024-08-01T18:59:42.068Z pid=1315 tid=cf INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-hosts-queue-1[1313]: 2024-08-01T18:59:42.070Z pid=1313 tid=cd INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@orchestrator[1311]: 2024-08-01T18:59:42.068Z pid=1311 tid=ar INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-hosts-queue-1[1313]: 2024-08-01T18:59:42.073Z pid=1313 tid=cd INFO: GitLab reliable fetch activated!
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@orchestrator[1311]: 2024-08-01T18:59:42.073Z pid=1311 tid=ar INFO: GitLab reliable fetch activated!
Aug 01 14:59:42 themis.cecnet.gmu.edu dynflow-sidekiq@worker-1[1315]: 2024-08-01T18:59:42.073Z pid=1315 tid=cf INFO: GitLab reliable fetch activated!
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-2[1218]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-3[1214]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-1[1213]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 01 15:00:00 themis.cecnet.gmu.edu pulpcore-worker-4[1216]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
22

ACTUALLY…

It looks like pulpcore-api is not working correctly:

Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4124]: [2024-08-01 19:27:59 +0000] [4124] [INFO] Starting gunicorn 20.1.0
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4124]: [2024-08-01 19:27:59 +0000] [4124] [INFO] Listening at: unix:/run/pulpcore-api.sock (4124)
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4124]: [2024-08-01 19:27:59 +0000] [4124] [INFO] Using worker: pulpcore.app.entrypoint.PulpApiWorker
Aug 01 15:27:59 themis.cecnet.gmu.edu systemd[1]: Started Pulp API Server.
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4136]: [2024-08-01 19:27:59 +0000] [4136] [INFO] Booting worker with pid: 4136
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4137]: [2024-08-01 19:27:59 +0000] [4137] [INFO] Booting worker with pid: 4137
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4138]: [2024-08-01 19:27:59 +0000] [4138] [INFO] Booting worker with pid: 4138
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4139]: [2024-08-01 19:27:59 +0000] [4139] [INFO] Booting worker with pid: 4139
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4140]: [2024-08-01 19:27:59 +0000] [4140] [INFO] Booting worker with pid: 4140
Aug 01 15:27:59 themis.cecnet.gmu.edu pulpcore-api[4140]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4140@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4136]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4136@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4137]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4137@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4138]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4138@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
Aug 01 15:28:44 themis.cecnet.gmu.edu pulpcore-api[4139]: pulp [None]: pulpcore.app.entrypoint:INFO: Api App '4139@themis.cecnet.gmu.edu' failed to write a heartbeat to the database, sleeping for '45.0' seconds.
23

That is a different issue. Please open a separate thread or search if another thread covers it already. This thread is about an issue upgrading from 3.10 to 3.11.

24

Or, I could feel unwelcome and just go back to lurking.

25

You are not un-welcome, it just really looks like a different issue, to not clutter the thread it would be the best to move this to it’s own thread :slightly_smiling_face:
(and if it’s in the end the same issue, finding out also doesn’t clutter the main thread)