Pulpcore module is missing

Support
,
1

Problem: After upgrading katello version to 4.14 i had some issues /conflicts with pulp core along with python 39so i thought will reset it but did not help then i did remove the pulpcore but unfortunately now even after i install with python3.11 it is not available as module when i list it hence i am unable to enable it . Eventually my pulp core api and pulp worker services are failing. Can someone help me here

Expected outcome:

Foreman and Proxy versions: 3.12.1

Foreman and Proxy plugin versions: 3.12.1

Distribution and version:

Other relevant data:


ulpcore-content.service - Pulp Content App
   Loaded: loaded (/etc/systemd/system/pulpcore-content.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-12-18 12:51:35 UTC; 5min ago
 Main PID: 108437 (pulpcore-conten)
   Status: "Gunicorn arbiter booted"
    Tasks: 28 (limit: 152216)
   Memory: 363.3M
   CGroup: /system.slice/pulpcore-content.service
           ├─108437 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108482 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108485 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108488 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108493 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108496 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108499 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108502 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           ├─108505 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -
           └─108508 /usr/bin/python3.11 /usr/bin/pulpcore-content --preload --timeout 90 --workers 9 --access-logfile -

Dec 18 12:55:23 katello pulpcore-content[108505]:  [18/Dec/2024:12:55:23 +0000] "GET /pulp/content/xxx/Library/custom/Percona-Tools/Percona_Tools_-_EL8/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.10; generic; Linux.x86_64)"
Dec 18 12:57:18 katello pulpcore-content[108505]:  [18/Dec/2024:12:57:18 +0000] "GET /pulp/content/xxx/Library/custom/Rocky_Linux_8/Rocky_Linux_8_-_AppStream/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:18 katello pulpcore-content[108505]:  [18/Dec/2024:12:57:18 +0000] "GET /pulp/content/xxx/Library/custom/Jenkins/Jenkins/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:18 katello pulpcore-content[108499]:  [18/Dec/2024:12:57:18 +0000] "GET /pulp/content/xxx/Library/custom/Google_Cloud_-_EL8/Google_Cloud_SDK_-_EL8/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:18 katello pulpcore-content[108493]:  [18/Dec/2024:12:57:18 +0000] "GET /pulp/content/xxx/Library/custom/Remi_EL8/Remi_EL8_-_PHP_7_4/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:18 katello pulpcore-content[108488]:  [18/Dec/2024:12:57:18 +0000] "GET /pulp/content/xxx/Library/custom/EPEL-8/EPEL-8/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:19 katello pulpcore-content[108505]:  [18/Dec/2024:12:57:19 +0000] "GET /pulp/content/xxx/Library/custom/Grafana/Grafana/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:19 katello pulpcore-content[108485]:  [18/Dec/2024:12:57:19 +0000] "GET /pulp/content/xxx/Library/custom/Docker_CE_-_EL8/Docker_CE_-_EL8/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:19 katello pulpcore-content[108505]:  [18/Dec/2024:12:57:19 +0000] "GET /pulp/content/xxx/Library/custom/InfluxDB_-_EL8/InfluxDB_-_EL8/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
Dec 18 12:57:19 katello pulpcore-content[108502]:  [18/Dec/2024:12:57:19 +0000] "GET /pulp/content/xxx/Library/custom/Rocky_Linux_8/Rocky_Linux_8_7_-_AppStream/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Rocky Linux 8.8; generic; Linux.x86_64)"
/ displaying pulpcore-worker@1.service
â—Ź pulpcore-worker@1.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2024-12-18 12:57:30 UTC; 1s ago
  Process: 110026 ExecStart=/usr/libexec/pulpcore/pulpcore-worker (code=exited, status=1/FAILURE)
 Main PID: 110026 (code=exited, status=1/FAILURE)

Dec 18 12:57:30 katello systemd[1]: pulpcore-worker@1.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 12:57:30 katello systemd[1]: pulpcore-worker@1.service: Failed with result 'exit-code'.
/ displaying pulpcore-worker@2.service
â—Ź pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2024-12-18 12:57:30 UTC; 1s ago
  Process: 110027 ExecStart=/usr/libexec/pulpcore/pulpcore-worker (code=exited, status=1/FAILURE)
 Main PID: 110027 (code=exited, status=1/FAILURE)

Am kind of new to it . can someone help me here

2

image
image1346Ă—636 22.6 KB

3

I suggest you check the installation docs, make sure that all necessary repositories are enabled, (re)install the packages again and run foreman-installer. That should fix everything again. If not, you would need to post which errors the installer shows.

4

Hello gvde,

thanks for getting back

2024-12-18 13:51:10 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-12-18 13:51:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-12-18 13:51:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-12-18 13:51:19 [NOTICE] [configure] Starting system configuration.
2024-12-18 13:51:31 [NOTICE] [configure] 250 configuration steps out of 1784 steps complete.
2024-12-18 13:51:32 [NOTICE] [configure] 500 configuration steps out of 1786 steps complete.
2024-12-18 13:51:35 [NOTICE] [configure] 750 configuration steps out of 1789 steps complete.
2024-12-18 13:51:35 [NOTICE] [configure] 1000 configuration steps out of 1795 steps complete.
2024-12-18 13:51:35 [NOTICE] [configure] 1250 configuration steps out of 1795 steps complete.
2024-12-18 13:51:36 [NOTICE] [configure] 1500 configuration steps out of 1795 steps complete.
2024-12-18 13:52:05 [ERROR ] [configure] Systemd start for pulpcore-api.socket failed!
2024-12-18 13:52:05 [ERROR ] [configure] journalctl log for pulpcore-api.socket:
2024-12-18 13:52:05 [ERROR ] [configure] -- Logs begin at Wed 2024-12-18 09:16:58 UTC, end at Wed 2024-12-18 13:52:05 UTC. --
2024-12-18 13:52:05 [ERROR ] [configure] Dec 18 13:52:05 katello systemd[1]: pulpcore-api.socket: Socket service pulpcore-api.service already active, refusing.
2024-12-18 13:52:05 [ERROR ] [configure] Dec 18 13:52:05 katello systemd[1]: Failed to listen on Pulp API Server socket.
2024-12-18 13:52:05 [ERROR ] [configure] /Stage[main]/Pulpcore::Service/Pulpcore::Socket_service[pulpcore-api]/Systemd::Unit_file[pulpcore-api.socket]/Service[pulpcore-api.socket]/ensure: change from 'stopped' to 'running' failed: Systemd start for pulpcore-api.socket failed!
2024-12-18 13:52:05 [ERROR ] [configure] journalctl log for pulpcore-api.socket:
2024-12-18 13:52:05 [ERROR ] [configure] -- Logs begin at Wed 2024-12-18 09:16:58 UTC, end at Wed 2024-12-18 13:52:05 UTC. --
2024-12-18 13:52:05 [ERROR ] [configure] Dec 18 13:52:05 katello systemd[1]: pulpcore-api.socket: Socket service pulpcore-api.service already active, refusing.
2024-12-18 13:52:05 [ERROR ] [configure] Dec 18 13:52:05 katello systemd[1]: Failed to listen on Pulp API Server socket.
2024-12-18 13:52:05 [NOTICE] [configure] 1750 configuration steps out of 1795 steps complete.
2024-12-18 13:52:09 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Service resource 'pulpcore-api.socket' failed. Logs:
  /Service[pulpcore-api.socket]
    Starting to evaluate the resource (1709 of 1795)
    Evaluated in 0.27 seconds
  /Stage[main]/Pulpcore::Service/Pulpcore::Socket_service[pulpcore-api]/Systemd::Unit_file[pulpcore-api.socket]/Service[pulpcore-api.socket]/ensure
    change from 'stopped' to 'running' failed: Systemd start for pulpcore-api.socket failed!
journalctl log for pulpcore-api.socket:
-- Logs begin at Wed 2024-12-18 09:16:58 UTC, end at Wed 2024-12-18 13:52:05 UTC. --
Dec 18 13:52:05 katello systemd[1]: pulpcore-api.socket: Socket service pulpcore-api.service already active, refusing.
Dec 18 13:52:05 katello systemd[1]: Failed to listen on Pulp API Server socket.

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.
5

I did a restart of svc and it seems to be active


[root@katello ~]# systemctl status pulpcore-api.socket
â—Ź pulpcore-api.socket - Pulp API Server socket
   Loaded: loaded (/etc/systemd/system/pulpcore-api.socket; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2024-12-18 12:51:30 UTC; 1h 2min ago
   Listen: /run/pulpcore-api.sock (Stream)

Dec 18 09:17:13 katello systemd[1]: Listening on Pulp API Server socket.
Dec 18 12:51:30 katello systemd[1]: pulpcore-api.socket: Succeeded.
Dec 18 12:51:30 katello systemd[1]: Closed Pulp API Server socket.
Dec 18 12:51:30 katello systemd[1]: Stopping Pulp API Server socket.
Dec 18 12:51:30 katello systemd[1]: pulpcore-api.socket: Socket service pulpcore-api.service already active, refusing.
Dec 18 12:51:30 katello systemd[1]: Failed to listen on Pulp API Server socket.
Dec 18 12:51:31 katello systemd[1]: pulpcore-api.socket: Socket service pulpcore-api.service already active, refusing.
Dec 18 12:51:31 katello systemd[1]: Failed to listen on Pulp API Server socket.
Dec 18 13:52:05 katello systemd[1]: pulpcore-api.socket: Socket service pulpcore-api.service already active, refusing.
Dec 18 13:52:05 katello systemd[1]: Failed to listen on Pulp API Server socket.
[root@katello ~]# systemctl restart pulpcore-api.socket
[root@katello ~]# systemctl status pulpcore-api.socket
â—Ź pulpcore-api.socket - Pulp API Server socket
   Loaded: loaded (/etc/systemd/system/pulpcore-api.socket; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-12-18 13:54:28 UTC; 3s ago
   Listen: /run/pulpcore-api.sock (Stream)
    Tasks: 0 (limit: 152216)
   Memory: 4.0K
   CGroup: /system.slice/pulpcore-api.socket

Dec 18 13:54:28 katello systemd[1]: Starting Pulp API Server socket.
Dec 18 13:54:28 katello systemd[1]: Listening on Pulp API Server socket.

but now after this has become active i ran the foreman-installer again and it turns to be successful but unfortunately when i check the health check

[root@katello pgsql]# foreman-maintain health check --label services-up
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------

[root@katello pgsql]# foreman-maintain health check --label services-up
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check whether all services are running:                               [FAIL]
Following services are not running: pulpcore-worker@1.service, pulpcore-worker@2.service
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)]

also in foreman-maintain service status also it shows the pulp core as failed and still am unable to find the pulpcore module too.

it shows like above also in katello page shows

image
image1346Ă—636 22.8 KB

6

There seems to be some processes still running when you started the installer. Make sure to stop foreman before running foreman-installer

# foreman-maintain service stop

Check with ps that really everything has been stopped. Then run foreman-installer. If there are still errors, please check the production.log for errors.

That should just check the systemd services. Where do you see “pulp core” as failed there?

I don’t know what you are looking for. There is no pulpcore module in the service status.

# dnf reinstall foreman-installer-katello

runs without errors?

7

I just ran

foreman-maintain service stop and then i checked with ps

[root@katello ~]# ps
    PID TTY          TIME CMD
 282982 pts/0    00:00:00 sudo
 282992 pts/0    00:00:00 bash
 283555 pts/0    00:00:00 ps

and when i run foreman-installer i get

2024-12-19 07:04:03 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-12-19 07:04:06 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-12-19 07:04:06 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-12-19 07:04:12 [NOTICE] [configure] Starting system configuration.
2024-12-19 07:04:23 [NOTICE] [configure] 250 configuration steps out of 1784 steps complete.
2024-12-19 07:04:24 [NOTICE] [configure] 500 configuration steps out of 1786 steps complete.
2024-12-19 07:04:27 [NOTICE] [configure] 750 configuration steps out of 1789 steps complete.
2024-12-19 07:04:27 [NOTICE] [configure] 1000 configuration steps out of 1795 steps complete.
2024-12-19 07:04:27 [NOTICE] [configure] 1250 configuration steps out of 1795 steps complete.
2024-12-19 07:04:47 [NOTICE] [configure] 1500 configuration steps out of 1795 steps complete.
2024-12-19 07:05:36 [ERROR ] [configure] Could not find command 'pulpcore-manager'
2024-12-19 07:05:36 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: Could not find command 'pulpcore-manager'
2024-12-19 07:06:31 [NOTICE] [configure] 1750 configuration steps out of 1795 steps complete.
2024-12-19 07:06:34 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Exec resource 'pulpcore-manager migrate --noinput' failed. Logs:
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]
    Adding autorequire relationship with File[/var/lib/pulp]
    Adding autorequire relationship with User[pulp]
    Starting to evaluate the resource (1640 of 1795)
    Evaluated in 0.00 seconds
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns
    change from 'notrun' to ['0'] failed: Could not find command 'pulpcore-manager'

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

Also when i say pulpcore is not available i mean

[root@katello ~]# sudo dnf module list pulpcore
Last metadata expiration check: 1:11:24 ago on Thu 19 Dec 2024 06:04:45 AM UTC.
Error: No matching Modules to list

[root@katello ~]# sudo dnf module enable pulpcore:el8
Last metadata expiration check: 1:11:55 ago on Thu 19 Dec 2024 06:04:45 AM UTC.
Error: Problems in request:
missing groups or modules: pulpcore:el8

And when i check the service status by foreman-maintain service status

displaying pulpcore-api
â—Ź pulpcore-api.service - Pulp API Server
   Loaded: loaded (/etc/systemd/system/pulpcore-api.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2024-12-19 06:59:08 UTC; 18min ago
 Main PID: 127834 (code=exited, status=0/SUCCESS)
   Status: "Gunicorn arbiter booted"

Dec 19 06:59:05 katello systemd[1]: Stopping Pulp API Server...
Dec 19 06:59:05 katello pulpcore-api[127834]: [2024-12-19 06:59:05 +0000] [127834] [INFO] Handling signal: term
Dec 19 06:59:05 katello pulpcore-api[127877]: [2024-12-19 06:59:05 +0000] [127877] [INFO] Worker exiting (pid: 127877)
Dec 19 06:59:05 katello pulpcore-api[127890]: [2024-12-19 06:59:05 +0000] [127890] [INFO] Worker exiting (pid: 127890)
Dec 19 06:59:05 katello pulpcore-api[127895]: [2024-12-19 06:59:05 +0000] [127895] [INFO] Worker exiting (pid: 127895)
Dec 19 06:59:05 katello pulpcore-api[127889]: [2024-12-19 06:59:05 +0000] [127889] [INFO] Worker exiting (pid: 127889)
Dec 19 06:59:05 katello pulpcore-api[128024]: [2024-12-19 06:59:05 +0000] [128024] [INFO] Worker exiting (pid: 128024)
Dec 19 06:59:07 katello pulpcore-api[127834]: [2024-12-19 06:59:07 +0000] [127834] [INFO] Shutting down: Master
Dec 19 06:59:08 katello systemd[1]: pulpcore-api.service: Succeeded.
Dec 19 06:59:08 katello systemd[1]: Stopped Pulp API Server.
/ displaying pulpcore-content
â—Ź pulpcore-content.service - Pulp Content App
   Loaded: loaded (/etc/systemd/system/pulpcore-content.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2024-12-19 06:59:08 UTC; 18min ago
 Main PID: 127830 (code=exited, status=0/SUCCESS)
   Status: "Gunicorn arbiter booted"

Dec 19 06:59:05 katello pulpcore-content[127863]: [2024-12-19 06:59:05 +0000] [127863] [INFO] Worker exiting (pid: 127863)
Dec 19 06:59:05 katello pulpcore-content[127851]: [2024-12-19 06:59:05 +0000] [127851] [INFO] Worker exiting (pid: 127851)
Dec 19 06:59:05 katello pulpcore-content[127860]: [2024-12-19 06:59:05 +0000] [127860] [INFO] Worker exiting (pid: 127860)
Dec 19 06:59:05 katello pulpcore-content[127865]: [2024-12-19 06:59:05 +0000] [127865] [INFO] Worker exiting (pid: 127865)
Dec 19 06:59:05 katello pulpcore-content[127867]: [2024-12-19 06:59:05 +0000] [127867] [INFO] Worker exiting (pid: 127867)
Dec 19 06:59:06 katello pulpcore-content[127858]: [2024-12-19 06:59:06 +0000] [127858] [INFO] Worker exiting (pid: 127858)
Dec 19 06:59:06 katello pulpcore-content[127846]: sys:1: RuntimeWarning: coroutine 'Connection.disconnect' was never awaited
Dec 19 06:59:07 katello pulpcore-content[127830]: [2024-12-19 06:59:07 +0000] [127830] [INFO] Shutting down: Master
Dec 19 06:59:08 katello systemd[1]: pulpcore-content.service: Succeeded.
Dec 19 06:59:08 katello systemd[1]: Stopped Pulp Content App.
/ displaying pulpcore-worker@1.service
â—Ź pulpcore-worker@1.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: inactive (dead) (Result: exit-code) since Thu 2024-12-19 06:59:05 UTC; 18min ago
 Main PID: 283369 (code=exited, status=1/FAILURE)

Dec 19 06:59:05 katello systemd[1]: Stopped Pulp Worker.
/ displaying pulpcore-worker@2.service
â—Ź pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: inactive (dead) (Result: exit-code) since Thu 2024-12-19 06:59:05 UTC; 18min ago
 Main PID: 283407 (code=exited, status=1/FAILURE)

Dec 19 06:59:05 katello systemd[1]: pulpcore-worker@2.service: Main process exited, code=exited, status=1/FAILURE
Dec 19 06:59:05 katello systemd[1]: pulpcore-worker@2.service: Failed with result 'exit-code'.
Dec 19 06:59:05 katello systemd[1]: Stopped Pulp Worker.

- All services displayed                                              [FAIL]
Some services are not running (pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, foreman-proxy)
8

I have meant: check with ps if there are any foreman/pulpcore processes running. Of course, you have to use ps -ef.

Again: you have to post the errors in the production log /var/log/foreman/production.log.

Again: please check and read the docs. The 3.12/4/14 installation docs don’t say about enabling or using the pulpcore dnf module. It only tells you to enable the katello:el8 module and only on el8 (which is deprecated and 3.12/4.14 is the final version supporting el8).

When upgrading you really have to follow the docs, too. The docs for upgrading to 4.14 Upgrading Foreman to 3.12 tell you to disable the pulpcore module in case it is still enabled, because it has been removed with 3.9/4.11 and when you have upgraded to 3.9/4.11 following the 4.11 upgrade docs you would have done so at that time.

The upgrade docs unfortunately don’t fully cover the transition to el9, yet, thus it contains some steps like dnf module enable katello:el8 which of course don’t apply if you are already running on el9.

You didn’t answer which distro and version you are using, not even if you are on el8 or el9.

Something off. You must have removed the pulpcore packages from the system. What do you have installed:

# rpm -qa '*pulp*' '*katello*' '*foreman*' | sort

Did you reinstall as I suggested before?

# dnf reinstall foreman-installer-katello
9

We are using el8 . sorry am not that expert with katello part should have shared it earlier itself.

[root@katello ~]# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 Dec18 ?        00:04:04 /usr/lib/systemd/systemd --system --deserialize 19
root           2       0  0 Dec18 ?        00:00:00 [kthreadd]
root           3       2  0 Dec18 ?        00:00:00 [rcu_gp]
root           4       2  0 Dec18 ?        00:00:00 [rcu_par_gp]
root           5       2  0 Dec18 ?        00:00:00 [slub_flushwq]
root           7       2  0 Dec18 ?        00:00:00 [kworker/0:0H-events_highpri]
root          10       2  0 Dec18 ?        00:00:00 [mm_percpu_wq]
root          11       2  0 Dec18 ?        00:00:00 [rcu_tasks_rude_]
root          12       2  0 Dec18 ?        00:00:00 [rcu_tasks_trace]
root          13       2  0 Dec18 ?        00:00:03 [ksoftirqd/0]
root          14       2  0 Dec18 ?        00:00:55 [rcu_sched]
root          15       2  0 Dec18 ?        00:00:00 [migration/0]
root          16       2  0 Dec18 ?        00:00:00 [watchdog/0]
root          17       2  0 Dec18 ?        00:00:00 [cpuhp/0]
root          18       2  0 Dec18 ?        00:00:00 [cpuhp/1]
root          19       2  0 Dec18 ?        00:00:00 [watchdog/1]
root          20       2  0 Dec18 ?        00:00:00 [migration/1]
root          21       2  0 Dec18 ?        00:00:02 [ksoftirqd/1]
root          23       2  0 Dec18 ?        00:00:00 [kworker/1:0H-events_highpri]
root          24       2  0 Dec18 ?        00:00:00 [cpuhp/2]
root          25       2  0 Dec18 ?        00:00:00 [watchdog/2]
root          26       2  0 Dec18 ?        00:00:00 [migration/2]
root          27       2  0 Dec18 ?        00:00:02 [ksoftirqd/2]
root          29       2  0 Dec18 ?        00:00:00 [kworker/2:0H-events_highpri]
root          30       2  0 Dec18 ?        00:00:00 [cpuhp/3]
root          31       2  0 Dec18 ?        00:00:00 [watchdog/3]
root          32       2  0 Dec18 ?        00:00:00 [migration/3]
root          33       2  0 Dec18 ?        00:00:03 [ksoftirqd/3]
root          35       2  0 Dec18 ?        00:00:00 [kworker/3:0H-events_highpri]
root          40       2  0 Dec18 ?        00:00:00 [kdevtmpfs]
root          41       2  0 Dec18 ?        00:00:00 [netns]
root          42       2  0 Dec18 ?        00:00:01 [kauditd]
root          43       2  0 Dec18 ?        00:00:00 [khungtaskd]
root          44       2  0 Dec18 ?        00:00:00 [oom_reaper]
root          45       2  0 Dec18 ?        00:00:00 [writeback]
root          46       2  0 Dec18 ?        00:00:00 [kcompactd0]
root          47       2  0 Dec18 ?        00:00:00 [ksmd]
root          48       2  0 Dec18 ?        00:00:02 [khugepaged]
root          49       2  0 Dec18 ?        00:00:00 [crypto]
root          50       2  0 Dec18 ?        00:00:00 [kintegrityd]
root          51       2  0 Dec18 ?        00:00:00 [kblockd]
root          52       2  0 Dec18 ?        00:00:00 [blkcg_punt_bio]
root          54       2  0 Dec18 ?        00:00:00 [tpm_dev_wq]
root          55       2  0 Dec18 ?        00:00:00 [md]
root          56       2  0 Dec18 ?        00:00:00 [md_bitmap]
root          57       2  0 Dec18 ?        00:00:00 [edac-poller]
root          58       2  0 Dec18 ?        00:00:00 [watchdogd]
root          60       2  0 Dec18 ?        00:00:02 [kworker/3:1H-xfs-log/sda2]
root          77       2  0 Dec18 ?        00:00:00 [kswapd0]
root         138       2  0 Dec18 ?        00:00:00 [kthrotld]
root         139       2  0 Dec18 ?        00:00:00 [acpi_thermal_pm]
root         140       2  0 Dec18 ?        00:00:00 [hwrng]
root         141       2  0 Dec18 ?        00:00:00 [kmpath_rdacd]
root         142       2  0 Dec18 ?        00:00:00 [kaluad]
root         144       2  0 Dec18 ?        00:00:00 [ipv6_addrconf]
root         145       2  0 Dec18 ?        00:00:00 [kstrp]
root         185       2  0 Dec18 ?        00:00:00 [zswap-shrink]
root         234       2  0 Dec18 ?        00:00:01 [kworker/1:1H-kblockd]
root         236       2  0 Dec18 ?        00:00:01 [kworker/2:1H-kblockd]
root         242       2  0 Dec18 ?        00:00:01 [kworker/0:1H-kblockd]
root         335       2  0 Dec18 ?        00:00:00 [nvme-wq]
root         336       2  0 Dec18 ?        00:00:00 [nvme-reset-wq]
root         337       2  0 Dec18 ?        00:00:00 [nvme-delete-wq]
root         391       2  0 Dec18 ?        00:00:00 [scsi_eh_0]
root         394       2  0 Dec18 ?        00:00:00 [scsi_tmf_0]
root         474       2  0 Dec18 ?        00:00:00 [xfsalloc]
root         475       2  0 Dec18 ?        00:00:00 [xfs_mru_cache]
root         476       2  0 Dec18 ?        00:00:00 [xfs-buf/sda2]
root         477       2  0 Dec18 ?        00:00:00 [xfs-conv/sda2]
root         478       2  0 Dec18 ?        00:00:00 [xfs-cil/sda2]
root         479       2  0 Dec18 ?        00:00:00 [xfs-reclaim/sda]
root         480       2  0 Dec18 ?        00:00:00 [xfs-blockgc/sda]
root         481       2  0 Dec18 ?        00:00:00 [xfs-log/sda2]
root         482       2  0 Dec18 ?        00:00:08 [xfsaild/sda2]
root         573       1  0 Dec18 ?        00:03:52 /usr/lib/systemd/systemd-journald
root         609       1  0 Dec18 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root         634       2  0 Dec18 ?        00:00:01 [jbd2/sdc-8]
root         636       2  0 Dec18 ?        00:00:00 [ext4-rsv-conver]
root         637       2  0 Dec18 ?        00:00:00 [jbd2/sdd-8]
root         638       2  0 Dec18 ?        00:00:00 [ext4-rsv-conver]
root         639       2  0 Dec18 ?        00:00:01 [jbd2/sdb-8]
root         640       2  0 Dec18 ?        00:00:00 [ext4-rsv-conver]
root         681       2  0 Dec18 ?        00:00:00 [nfit]
root         720       1  0 Dec18 ?        00:00:12 /sbin/auditd
daemon       755       1  0 Dec18 ?        00:00:19 /usr/sbin/rngd -f --fill-watermark=0 -x pkcs11 -x nist -x qrypt -D daemon:daemon
dbus         765       1  0 Dec18 ?        00:01:34 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         767       1  0 Dec18 ?        00:00:00 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid
chrony       769       1  0 Dec18 ?        00:00:00 /usr/sbin/chronyd
root         770       1  0 Dec18 ?        00:00:00 /usr/sbin/saslauthd -m /run/saslauthd -a pam
root         771     770  0 Dec18 ?        00:00:00 /usr/sbin/saslauthd -m /run/saslauthd -a pam
root         772     770  0 Dec18 ?        00:00:00 /usr/sbin/saslauthd -m /run/saslauthd -a pam
root         773     770  0 Dec18 ?        00:00:00 /usr/sbin/saslauthd -m /run/saslauthd -a pam
root         774     770  0 Dec18 ?        00:00:00 /usr/sbin/saslauthd -m /run/saslauthd -a pam
root         785       1  0 Dec18 ?        00:00:02 /usr/sbin/NetworkManager --no-daemon
root         805       1  0 Dec18 ?        00:02:49 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
prometh+     818       1  0 Dec18 ?        00:01:08 /opt/prometheus/node_exporter-1.7.0.linux-amd64/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
root         840       1  0 Dec18 tty1     00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root         841       1  0 Dec18 ttyS0    00:00:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
root         961       1  0 Dec18 ?        00:00:00 /usr/libexec/postfix/master -w
postfix      963     961  0 Dec18 ?        00:00:00 qmgr -l -t unix -u
polkitd     1238       1  0 Dec18 ?        00:00:13 /usr/lib/polkit-1/polkitd --no-debug
root        1327       1  0 Dec18 ?        00:03:13 /usr/sbin/rsyslogd -n
root        1328       1  0 Dec18 ?        00:00:17 /usr/bin/google_osconfig_agent
root        1579       1  0 Dec18 ?        00:00:05 /usr/bin/google_guest_agent
root        1845       1  1 Dec18 ?        00:14:39 /usr/bin/google_guest_agent_manager
root        1852       1  0 Dec18 ?        00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-e
root        1886       1  0 Dec18 ?        00:00:18 /usr/lib/systemd/systemd-logind
root        1893       1  0 Dec18 ?        00:00:00 /usr/sbin/crond -n
kci        67393       1  0 Dec18 ?        00:00:06 /usr/lib/systemd/systemd --user
kci        67396   67393  0 Dec18 ?        00:00:00 (sd-pam)
root      128889       1  0 Dec18 ?        00:00:02 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent
root      273926       2  0 05:59 ?        00:00:00 [kworker/u8:2-events_unbound]
root      281471       2  0 06:49 ?        00:00:00 [kworker/u8:1-events_unbound]
postfix   282877     961  0 06:58 ?        00:00:00 pickup -l -t unix -u
root      282929    1852  0 06:58 ?        00:00:00 sshd: kci [priv]
kci       282934  282929  0 06:58 ?        00:00:00 sshd: kci@pts/0
kci       282936  282934  0 06:58 pts/0    00:00:00 -bash
root      282982  282936  0 06:58 pts/0    00:00:00 sudo -i
root      282992  282982  0 06:58 pts/0    00:00:00 -bash
root      295347       2  0 07:47 ?        00:00:00 [kworker/u8:0-events_unbound]
root      304185       2  0 08:18 ?        00:00:00 [kworker/0:0-events]
root      304192       2  0 08:18 ?        00:00:00 [kworker/2:0-events]
root      304342       2  0 08:18 ?        00:00:00 [kworker/1:1-mm_percpu_wq]
root      304354       2  0 08:19 ?        00:00:00 [kworker/3:3-events]
root      305617       2  0 08:23 ?        00:00:00 [kworker/0:3-cgroup_pidlist_destroy]
root      305706       2  0 08:23 ?        00:00:00 [kworker/2:3-cgroup_pidlist_destroy]
root      305773       2  0 08:24 ?        00:00:00 [kworker/3:2-cgroup_pidlist_destroy]
root      305823       2  0 08:24 ?        00:00:00 [kworker/1:2-cgroup_pidlist_destroy]
root      308090       2  0 08:29 ?        00:00:00 [kworker/2:2-cgroup_pidlist_destroy]
root      308191       2  0 08:29 ?        00:00:00 [kworker/0:1-cgroup_pidlist_destroy]
root      308237       2  0 08:29 ?        00:00:00 [kworker/1:0-cgroup_pidlist_destroy]
root      308291       2  0 08:29 ?        00:00:00 [kworker/3:0-cgroup_pidlist_destroy]
root      309798       2  0 08:34 ?        00:00:00 [kworker/2:1-mm_percpu_wq]
root      309896    1852  0 08:34 ?        00:00:00 sshd: kci [priv]
kci       309904  309896  0 08:34 ?        00:00:00 sshd: kci@pts/1
kci       309907  309904  0 08:34 pts/1    00:00:00 -bash
root      309953       2  0 08:34 ?        00:00:00 [kworker/0:2-cgroup_pidlist_destroy]
root      309960  309907  0 08:34 pts/1    00:00:00 sudo -i
root      309967       2  0 08:34 ?        00:00:00 [kworker/1:3-mm_percpu_wq]
root      309988  309960  0 08:34 pts/1    00:00:00 -bash
root      310025       2  0 08:34 ?        00:00:00 [kworker/3:1-events_power_efficient]
root      310230  282992  0 08:35 pts/0    00:00:00 ps -ef

Exactly . IN my initial query i have mentioned that by mistake i have removed pulpcore package by sudo dnf remove python39-pulpcore since there was a python package conflict.

Yes i have done that and still the issue remains same .

2024-12-19 08:40:07 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: Could not find command 'pulpcore-manager'

unable to see any errors regarding pulp there. But few errors were popping up though

cat /var/log/foreman/production.log | grep error
2024-12-19T03:51:04 [I|app|e3715a0c]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.5ms | Allocations: 123)
2024-12-19T03:51:04 [I|app|e3715a0c]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.8ms | Allocations: 214)
2024-12-19T03:51:04 [I|app|fd8427bb]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.5ms | Allocations: 121)
2024-12-19T03:51:04 [I|app|fd8427bb]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.8ms | Allocations: 209)
2024-12-19T04:21:04 [I|app|7614cf4e]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 123)
2024-12-19T04:21:04 [I|app|7614cf4e]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 214)
2024-12-19T04:21:04 [I|app|f5770a02]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 124)
2024-12-19T04:21:04 [I|app|f5770a02]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.5ms | Allocations: 215)
2024-12-19T04:51:04 [I|app|44c20e91]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 123)
2024-12-19T04:51:04 [I|app|44c20e91]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.5ms | Allocations: 214)
2024-12-19T04:51:04 [I|app|d01552fc]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 124)
2024-12-19T04:51:04 [I|app|d01552fc]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 215)
2024-12-19T05:21:04 [I|app|096c3a80]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 124)
2024-12-19T05:21:04 [I|app|096c3a80]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 216)
2024-12-19T05:21:04 [I|app|1c1bc8b5]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 125)
2024-12-19T05:21:04 [I|app|1c1bc8b5]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.5ms | Allocations: 217)
2024-12-19T05:51:04 [I|app|9d447970]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 123)
2024-12-19T05:51:04 [I|app|9d447970]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.7ms | Allocations: 214)
2024-12-19T05:51:04 [I|app|0e41aaf8]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 121)
2024-12-19T05:51:04 [I|app|0e41aaf8]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 209)
2024-12-19T06:21:04 [I|app|b7ace7e3]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 123)
2024-12-19T06:21:04 [I|app|b7ace7e3]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 214)
2024-12-19T06:21:04 [I|app|9246240a]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 125)
2024-12-19T06:21:04 [I|app|9246240a]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 217)
2024-12-19T06:51:04 [I|app|3b237a44]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 123)
2024-12-19T06:51:04 [I|app|3b237a44]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.8ms | Allocations: 214)
2024-12-19T06:51:04 [I|app|64f64f7c]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 121)
2024-12-19T06:51:04 [I|app|64f64f7c]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 209)
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task 0b25248c-722b-4e16-a9c8-93e4f261a188' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea4031aeb8>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task 1c1b2e8d-d49c-43ad-9601-b8ecd9e8540f' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea3feb6390>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task 302b170a-b7d8-47d9-a2c7-c299093aa785' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea3fe8a7e0>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task 3ed255bf-ebba-4099-80f2-3563e52c8bea' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea3fe60648>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task 415a2bb5-ee25-402b-bdf0-b2dd44068d27' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea402c1f20>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task 6665facf-b489-405c-83d5-6651264e7413' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea402819c0>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task 784efde2-ddca-4560-b0c7-49f1a8f48f1c' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea400f9030>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task d0b35349-91f8-474c-8c94-39996b488869' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea400c4768>
2024-12-19T07:05:54 [I|for|] Backtrace for 'Failed at consistency check for task ef15fb04-039a-46d9-8efb-bf5182859440' error (NoMethodError): undefined method `id' for #<Hash:0x000055ea400786d8>
2024-12-19T07:06:10 [W|bac|59d7b83a] Error message: the server returns an error
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
2024-12-19T07:06:10 [E|bac|59d7b83a] Error message: the server returns an error
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
2024-12-19T07:06:10 [E|bac|59d7b83a] Error message: the server returns an error
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
2024-12-19T07:06:10 [E|bac|59d7b83a] Error message: the server returns an error
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
2024-12-19T07:06:10 [E|bac|59d7b83a] Error message: the server returns an error
 59d7b83a | Response body: {"displayMessage": "Service unavailable or restarting, try later", "errors": ["Service unavailable or restarting, try later"], "status": "503" } (PulpCertguardClient::ApiError)
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
2024-12-19T07:06:11 [E|bac|59d7b83a] Error message: the server returns an error
 59d7b83a | Response body: {"displayMessage": "Service unavailable or restarting, try later", "errors": ["Service unavailable or restarting, try later"], "status": "503" } (PulpCertguardClient::ApiError)
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
2024-12-19T07:06:11 [E|bac|59d7b83a] Error message: the server returns an error
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 59d7b83a | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
2024-12-19T07:21:01 [I|app|6284eb9a]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 1.0ms | Allocations: 268)
2024-12-19T07:21:01 [I|app|6284eb9a]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 1.8ms | Allocations: 485)
2024-12-19T07:21:05 [I|app|d64cfc68]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.9ms | Allocations: 268)
2024-12-19T07:21:05 [I|app|d64cfc68]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 1.5ms | Allocations: 485)
2024-12-19T07:21:06 [I|app|9efdf633]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 121)
2024-12-19T07:21:06 [I|app|9efdf633]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.7ms | Allocations: 210)
2024-12-19T07:25:47 [I|app|d96e008a]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 1.2ms | Allocations: 269)
2024-12-19T07:25:47 [I|app|d96e008a]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 2.0ms | Allocations: 486)
2024-12-19T07:40:09 [I|app|419232d4]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 124)
2024-12-19T07:40:09 [I|app|419232d4]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.7ms | Allocations: 215)
2024-12-19T07:49:58 [I|app|0cd05d36]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 1.3ms | Allocations: 269)
2024-12-19T07:49:58 [I|app|0cd05d36]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 2.0ms | Allocations: 488)
2024-12-19T07:51:01 [I|app|63acffca]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 123)
2024-12-19T07:51:01 [I|app|63acffca]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.7ms | Allocations: 215)
2024-12-19T07:51:05 [I|app|c197f76d]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 120)
2024-12-19T07:51:05 [I|app|c197f76d]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 208)
2024-12-19T07:51:05 [I|app|fbafcacc]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 121)
2024-12-19T07:51:05 [I|app|fbafcacc]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.7ms | Allocations: 209)
2024-12-19T08:21:04 [I|app|f9da0eed]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 124)
2024-12-19T08:21:04 [I|app|f9da0eed]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 216)
2024-12-19T08:21:04 [I|app|4ed94a12]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 121)
2024-12-19T08:21:04 [I|app|4ed94a12]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.6ms | Allocations: 209)
2024-12-19T08:26:41 [I|app|d710dec1]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 125)
2024-12-19T08:26:41 [I|app|d710dec1]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.7ms | Allocations: 217)
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task 0b25248c-722b-4e16-a9c8-93e4f261a188' error (NoMethodError): undefined method `id' for #<Hash:0x00005628e0ffb7c0>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task 1c1b2e8d-d49c-43ad-9601-b8ecd9e8540f' error (NoMethodError): undefined method `id' for #<Hash:0x00005628df594080>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task 302b170a-b7d8-47d9-a2c7-c299093aa785' error (NoMethodError): undefined method `id' for #<Hash:0x00005628dbe066e0>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task 3ed255bf-ebba-4099-80f2-3563e52c8bea' error (NoMethodError): undefined method `id' for #<Hash:0x00005628ed3e3430>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task 415a2bb5-ee25-402b-bdf0-b2dd44068d27' error (NoMethodError): undefined method `id' for #<Hash:0x00005628ed333760>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task 6665facf-b489-405c-83d5-6651264e7413' error (NoMethodError): undefined method `id' for #<Hash:0x00005628ed2ef268>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task 784efde2-ddca-4560-b0c7-49f1a8f48f1c' error (NoMethodError): undefined method `id' for #<Hash:0x00005628ed2c1930>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task d0b35349-91f8-474c-8c94-39996b488869' error (NoMethodError): undefined method `id' for #<Hash:0x00005628ed289580>
2024-12-19T08:40:27 [I|for|] Backtrace for 'Failed at consistency check for task ef15fb04-039a-46d9-8efb-bf5182859440' error (NoMethodError): undefined method `id' for #<Hash:0x00005628ed250a28>
10

One of the many reasons why I have asked for

# rpm -qa '*pulp*' '*katello*' '*foreman*' | sort

That is not the output of dnf reinstall foreman-installer-katello.

Did you only remove python39-pulpcore? Or something else, too? You can check with dnf history to see which packages you have touched.

Did you reinstall python39-pulpcore? So far it seems, the installer doesn’t reinstall it and the dependencies must be broken somewhere if foreman-installer-katello doesn’t install it again.

11

Got it ! Again running this does not return any output

[root@katello bin]# dnf history
ID     | Command line                                                                                                                                                            | Date and time    | Action(s)      | Altered
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   177 | reinstall foreman-installer-katello                                                                                                                                     | 2024-12-19 08:33 | R              |    2
   176 | install python3.11-pip                                                                                                                                                  | 2024-12-18 14:24 | Install        |    1
   175 | update                                                                                                                                                                  | 2024-12-18 13:33 | Upgrade        |    3
   174 | install pulpcore pulpcore-selinux python3.11-pulpcore                                                                                                                   | 2024-12-18 13:25 | Upgrade        |    1
   173 | install python3.11-pulpcore-3.49.21-1.el8.noarch.rpm                                                                                                                    | 2024-12-18 13:21 | Downgrade      |    4
   172 | -d 0 -e 1 -y upgrade redis-6.2.7-1.module+el8.7.0+1105+8815ce78                                                                                                         | 2024-12-18 12:49 | Upgrade        |    1 EE
   171 | -d 0 -e 1 -y install pulpcore-plugin(python)                                                                                                                            | 2024-12-18 12:46 | Install        |    4
   170 | -d 0 -e 1 -y install postgresql-contrib                                                                                                                                 | 2024-12-18 12:46 | Install        |    2
   169 | -d 0 -e 1 -y install foreman-redis                                                                                                                                      | 2024-12-18 12:46 | Install        |    1
   168 | -d 0 -e 1 -y install pulpcore-selinux                                                                                                                                   | 2024-12-18 12:40 | Install        |    1
   167 | update                                                                                                                                                                  | 2024-12-18 08:57 | Upgrade        |    5
   166 | install python3.11-pulpcore python3.11-pulp-rpm                                                                                                                         | 2024-12-18 08:55 | I, U           |    4
   165 | install python3.11-pulpcore-3.49.8-2.el8.noarch.rpm                                                                                                                     | 2024-12-18 08:32 | D, I           |    7
   164 | update --allowerasing                                                                                                                                                   | 2024-12-18 07:37 | C, E, I, O, U  |  223 EE
   163 | install python3.11-pulpcore                                                                                                                                             | 2024-12-18 07:34 | I, O           |  139
   162 | remove python39-pulpcore                                                                                                                                                | 2024-12-18 07:32 | Removed        |   29 EE

Yes reinstall completed without any issue later when i ran foreman-installer again i got the mentioned error .

Yes we tried and it did not work unfortunately.

12

If you are on katello 4.14 now, how did you get there? I start to suspect that you have skipped the version before and jumped from a much older version to 4.14. I think python 3.9 was used until katello 4.10 and when you have upgraded to 4.11 you would get pulpcore 3.39 using python3.11. It did so on my katello server. It automatically obsolete the python39-pulpcore package and removed it.

13

You are right!!! This was previously handled by one of my team mate who is not with us anymore so i had to update to lastest hence upgraded from 4.6 to 4.14 directly .

14

That is not supported. You have to upgrade through all intermediate versions following the respective upgrade docs.

You should revert the system or restore it from a backup and start over, upgrading from 3.6 to 3.7.

I can’t tell if the current system could be fully repaired, in particular because you have to do the leapp upgrade to el9 now which can be tricky and I don’t know if that’s a good idea if the system is already broken.

Or start a new system on el9.

Otherwise, maybe the developers have some suggestions what needs to be repaired to get the system properly running again.

15

That can’t be right. At least foreman-installer-katello must be listed. It’s really difficult, if you don’t post the exact commands you run and the output…

Did you disable the pulpcore module as the docs say?

Which repositories do you have enabled at the moment. Run

# dnf repolist -v
16 
[root@katello ~]# rpm -qa '*pulp*' '*katello*' '*foreman*' | sort
foreman-3.12.1-1.el8.noarch
foreman-cli-3.12.1-1.el8.noarch
foreman-debug-3.12.1-1.el8.noarch
foreman-dynflow-sidekiq-3.12.1-1.el8.noarch
foreman-installer-3.12.1-1.el8.noarch
foreman-installer-katello-3.12.1-1.el8.noarch
foreman-obsolete-packages-1.10-1.el8.noarch
foreman-postgresql-3.12.1-1.el8.noarch
foreman-proxy-3.12.1-1.el8.noarch
foreman-redis-3.12.1-1.el8.noarch
foreman-release-3.12.1-1.el8.noarch
foreman-selinux-3.12.1-1.el8.noarch
foreman-service-3.12.1-1.el8.noarch
katello-4.14.1-1.el8.noarch
katello-certs-tools-2.10.0-1.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.14.1-1.el8.noarch
katello-debug-4.14.1-1.el8.noarch
katello-default-ca-1.0-1.noarch
katello-repos-4.14.1-1.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
katello-server-ca-1.0-1.noarch
katello.kci.rocks-apache-1.0-1.noarch
katello.kci.rocks-foreman-client-1.0-1.noarch
katello.kci.rocks-foreman-proxy-1.0-1.noarch
katello.kci.rocks-foreman-proxy-client-1.0-1.noarch
katello.kci.rocks-puppet-client-1.0-1.noarch
pulp-client-1.0-1.noarch
pulpcore-obsolete-packages-1.2.0-1.el8.noarch
pulpcore-selinux-2.0.1-1.el8.x86_64
python3.11-pulp-cli-0.29.2-2.el8.noarch
python3.11-pulp-glue-0.29.2-2.el8.noarch
python3.11-pulp-python-3.11.3-1.el8.noarch
python3.11-pulp-rpm-3.26.1-1.el8.noarch
python3.11-pulpcore-3.49.22-1.el8.noarch
rubygem-foreman-tasks-9.2.3-1.fm3_12.el8.noarch
rubygem-foreman_maintain-1.7.6-1.el8.noarch
rubygem-foreman_remote_execution-13.2.5-1.fm3_12.el8.noarch
rubygem-hammer_cli_foreman-3.12.0-1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.21-1.fm3_11.el8.noarch
rubygem-hammer_cli_katello-1.14.3-1.el8.noarch
rubygem-katello-4.14.1-1.el8.noarch
rubygem-pulp_ansible_client-0.21.7-1.el8.noarch
rubygem-pulp_certguard_client-3.49.17-1.el8.noarch
rubygem-pulp_container_client-2.20.2-1.el8.noarch
rubygem-pulp_deb_client-3.2.1-1.el8.noarch
rubygem-pulp_file_client-3.49.17-1.el8.noarch
rubygem-pulp_ostree_client-2.3.2-1.el8.noarch
rubygem-pulp_python_client-3.11.2-1.el8.noarch
rubygem-pulp_rpm_client-3.26.1-1.el8.noarch
rubygem-pulpcore_client-3.49.17-1.el8.noarch
rubygem-smart_proxy_pulp-3.3.0-1.fm3_10.el8.noarch

[root@katello ~]# dnf repolist -v
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.7.0
cachedir: /var/cache/dnf
Last metadata expiration check: 1:02:55 ago on Thu 19 Dec 2024 09:55:30 AM UTC.
Repo-id            : appstream
Repo-name          : Rocky Linux 8 - AppStream
Repo-revision      : 8.10
Repo-distro-tags      : [cpe:/o:rocky:rocky:8]:  ,  , 8, L, R, c, i, k, n, o, u, x, y
Repo-updated       : Wed 18 Dec 2024 10:09:16 PM UTC
Repo-pkgs          : 9,883
Repo-available-pkgs: 7,974
Repo-size          : 24 G
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=AppStream-8
Repo-baseurl       : https://rocky-linux-europe-west1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//8.10/AppStream/x86_64/os/ (42 more)
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:21 AM UTC)
Repo-filename      : /etc/yum.repos.d/Rocky-AppStream.repo

Repo-id            : baseos
Repo-name          : Rocky Linux 8 - BaseOS
Repo-revision      : 8.10
Repo-distro-tags      : [cpe:/o:rocky:rocky:8]:  ,  , 8, L, R, c, i, k, n, o, u, x, y
Repo-updated       : Wed 18 Dec 2024 10:09:24 PM UTC
Repo-pkgs          : 2,879
Repo-available-pkgs: 2,875
Repo-size          : 6.2 G
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=BaseOS-8
Repo-baseurl       : https://rocky-linux-europe-west1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//8.10/BaseOS/x86_64/os/ (42 more)
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:24 AM UTC)
Repo-filename      : /etc/yum.repos.d/Rocky-BaseOS.repo

Repo-id            : candlepin
Repo-name          : Candlepin: an open source entitlement management system.
Repo-revision      : 1732539731
Repo-updated       : Mon 25 Nov 2024 01:02:11 PM UTC
Repo-pkgs          : 18
Repo-available-pkgs: 18
Repo-size          : 655 M
Repo-baseurl       : https://yum.theforeman.org/candlepin/4.4/el8/x86_64/
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:30 AM UTC)
Repo-filename      : /etc/yum.repos.d/katello.repo

Repo-id            : epel
Repo-name          : Extra Packages for Enterprise Linux 8 - x86_64
Repo-revision      : 1734583382
Repo-updated       : Thu 19 Dec 2024 04:43:58 AM UTC
Repo-pkgs          : 10,302
Repo-available-pkgs: 10,236
Repo-size          : 20 G
Repo-metalink      : https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=$infra&content=pub/rocky
  Updated          : Thu 19 Dec 2024 09:55:29 AM UTC
Repo-baseurl       : https://fedora.mirrorservice.org/epel/8/Everything/x86_64/ (298 more)
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:29 AM UTC)
Repo-exclude       : rubygem-net-ssh, rubygem-qpid_proton
Repo-filename      : /etc/yum.repos.d/epel.repo

Repo-id            : extras
Repo-name          : Rocky Linux 8 - Extras
Repo-revision      : 1734557872
Repo-distro-tags      : [cpe:/o:rocky:rocky:8]:  ,  , 8, L, R, c, i, k, n, o, u, x, y
Repo-updated       : Wed 18 Dec 2024 09:37:52 PM UTC
Repo-pkgs          : 57
Repo-available-pkgs: 57
Repo-size          : 3.3 M
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=extras-8
Repo-baseurl       : https://rocky-linux-europe-west1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//8.10/extras/x86_64/os/ (42 more)
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:27 AM UTC)
Repo-filename      : /etc/yum.repos.d/Rocky-Extras.repo

Repo-id            : foreman
Repo-name          : Foreman 3.12
Repo-revision      : 1732296882
Repo-updated       : Fri 22 Nov 2024 05:34:42 PM UTC
Repo-pkgs          : 638
Repo-available-pkgs: 638
Repo-size          : 336 M
Repo-baseurl       : https://yum.theforeman.org/releases/3.12/el8/x86_64
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:29 AM UTC)
Repo-filename      : /etc/yum.repos.d/foreman.repo

Repo-id            : foreman-plugins
Repo-name          : Foreman plugins 3.12
Repo-revision      : 1734557943
Repo-updated       : Wed 18 Dec 2024 09:39:03 PM UTC
Repo-pkgs          : 383
Repo-available-pkgs: 383
Repo-size          : 498 M
Repo-baseurl       : https://yum.theforeman.org/plugins/3.12/el8/x86_64
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:29 AM UTC)
Repo-filename      : /etc/yum.repos.d/foreman-plugins.repo

Repo-id            : google-cloud-sdk
Repo-name          : Google Cloud SDK
Repo-revision      : 1734452406391182
Repo-updated       : Mon 09 Feb 1925 09:58:38 PM UTC
Repo-pkgs          : 8,866
Repo-available-pkgs: 8,866
Repo-size          : 179 G
Repo-baseurl       : https://packages.cloud.google.com/yum/repos/cloud-sdk-el8-x86_64
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:29 AM UTC)
Repo-filename      : /etc/yum.repos.d/google-cloud.repo

Repo-id            : google-compute-engine
Repo-name          : Google Compute Engine
Repo-revision      : 1734382927726731
Repo-updated       : Wed 25 Feb 2037 06:58:19 AM UTC
Repo-pkgs          : 15
Repo-available-pkgs: 15
Repo-size          : 50 M
Repo-baseurl       : https://packages.cloud.google.com/yum/repos/google-compute-engine-el8-x86_64-stable
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:29 AM UTC)
Repo-filename      : /etc/yum.repos.d/google-cloud.repo

Repo-id            : katello
Repo-name          : Katello 4.14
Repo-revision      : 1731418583
Repo-updated       : Tue 12 Nov 2024 01:36:23 PM UTC
Repo-pkgs          : 75
Repo-available-pkgs: 75
Repo-size          : 213 M
Repo-baseurl       : https://yum.theforeman.org/katello/4.14/katello/el8/x86_64/
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:30 AM UTC)
Repo-filename      : /etc/yum.repos.d/katello.repo

Repo-id            : powertools
Repo-name          : Rocky Linux 8 - PowerTools
Repo-revision      : 8.10
Repo-distro-tags      : [cpe:/o:rocky:rocky:8]:  ,  , 8, L, R, c, i, k, n, o, u, x, y
Repo-updated       : Wed 18 Dec 2024 10:08:37 PM UTC
Repo-pkgs          : 2,931
Repo-available-pkgs: 2,717
Repo-size          : 39 G
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=PowerTools-8
Repo-baseurl       : https://rocky-linux-europe-west1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//8.10/PowerTools/x86_64/os/ (42 more)
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:27 AM UTC)
Repo-filename      : /etc/yum.repos.d/Rocky-PowerTools.repo

Repo-id            : pulpcore
Repo-name          : pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
Repo-revision      : 1729604810
Repo-updated       : Tue 22 Oct 2024 01:46:50 PM UTC
Repo-pkgs          : 357
Repo-available-pkgs: 357
Repo-size          : 112 M
Repo-baseurl       : https://yum.theforeman.org/pulpcore/3.49/el8/x86_64/
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:30 AM UTC)
Repo-filename      : /etc/yum.repos.d/katello.repo

Repo-id            : puppet7
Repo-name          : Puppet 7 Repository el 8 - x86_64
Repo-revision      : 1733225714
Repo-updated       : Tue 03 Dec 2024 11:35:14 AM UTC
Repo-pkgs          : 215
Repo-available-pkgs: 215
Repo-size          : 9.6 G
Repo-baseurl       : http://yum.puppet.com/puppet7/el/8/x86_64
Repo-expire        : 172,800 second(s) (last: Thu 19 Dec 2024 09:55:30 AM UTC)
Repo-filename      : /etc/yum.repos.d/puppet7-release.repo
Total packages: 36,619

unfortunately i removed it before itself.

17

I can only repeat that I would highly recommend to revert the server to the previous version and upgrade again. That is the safest way.

You should still disable it in case the information is still stored in the rpm database:

# dnf module disable pulpcore

Did you switch to postgresql:13 as required in the upgrade docs for 3.12?

You must not use EPEL! EPEL packages will interfere with foreman/katello. Disable the epel repository and check the installed packages (dnf list installed) for any packages installed from epel. Downgrade those packages if they are available through foreman/katello repositories.

To find broken rpm installations you can check with rpm -V:

# rpm -qa '*pulp*' '*katello*' '*foreman*'  | xargs rpm -V

The lines with the c are modified configuration files. Those are fine.

S.5....T.  c /etc/foreman-proxy/settings.d/pulpcore.yml

But in your case it should show /bin/pulpcore-manager as missing and possibly other issues.

18

yes i had done and it showed like


[root@katello ~]# dnf module disable pulpcore
Last metadata expiration check: 3:36:55 ago on Thu 19 Dec 2024 01:06:57 PM UTC.
Unable to resolve argument pulpcore
Error: Problems in request:
missing groups or modules: pulpcore


[root@katello ~]# rpm -qa '*pulp*' '*katello*' '*foreman*'  | xargs rpm -V
S.5....T.  c /etc/foreman-installer/scenarios.d/foreman-answers.yaml
S.5....T.  c /etc/foreman-installer/scenarios.d/foreman-migrations-applied
S.5....T.  c /etc/foreman-installer/scenarios.d/foreman.yaml
SM5...GT.  c /etc/foreman/database.yml
S.5...GT.  c /etc/foreman/dynflow/orchestrator.yml
missing   c /etc/foreman/dynflow/worker.yml
SM5...GT.  c /etc/foreman/settings.yaml
.M.......    /var/run/foreman
.......T.  c /etc/yum.repos.d/katello.repo
S.5....T.  c /etc/foreman-proxy/migration_state
S.5....T.  c /etc/foreman-proxy/settings.d/bmc.yml
S.5....T.  c /etc/foreman-proxy/settings.d/dhcp.yml
S.5....T.  c /etc/foreman-proxy/settings.d/dhcp_isc.yml
S.5....T.  c /etc/foreman-proxy/settings.d/dhcp_libvirt.yml
.......T.  c /etc/foreman-proxy/settings.d/dhcp_native_ms.yml
S.5....T.  c /etc/foreman-proxy/settings.d/dns.yml
.......T.  c /etc/foreman-proxy/settings.d/dns_dnscmd.yml
S.5....T.  c /etc/foreman-proxy/settings.d/dns_libvirt.yml
S.5....T.  c /etc/foreman-proxy/settings.d/dns_nsupdate.yml
S.5....T.  c /etc/foreman-proxy/settings.d/dns_nsupdate_gss.yml
.......T.  c /etc/foreman-proxy/settings.d/facts.yml
S.5....T.  c /etc/foreman-proxy/settings.d/httpboot.yml
..5....T.  c /etc/foreman-proxy/settings.d/logs.yml
.......T.  c /etc/foreman-proxy/settings.d/puppet.yml
S.5....T.  c /etc/foreman-proxy/settings.d/puppet_proxy_puppet_api.yml
S.5....T.  c /etc/foreman-proxy/settings.d/puppetca.yml
S.5....T.  c /etc/foreman-proxy/settings.d/puppetca_hostname_whitelisting.yml
S.5....T.  c /etc/foreman-proxy/settings.d/puppetca_http_api.yml
S.5....T.  c /etc/foreman-proxy/settings.d/puppetca_token_whitelisting.yml
.......T.  c /etc/foreman-proxy/settings.d/realm.yml
S.5....T.  c /etc/foreman-proxy/settings.d/realm_freeipa.yml
S.5....T.  c /etc/foreman-proxy/settings.d/registration.yml
S.5....T.  c /etc/foreman-proxy/settings.d/templates.yml
S.5....T.  c /etc/foreman-proxy/settings.d/tftp.yml
S.5....T.  c /etc/foreman-proxy/settings.yml
S.5....T.  c /etc/hammer/cli.modules.d/foreman.yml
missing     /usr/bin/pulp
S.5....T.  c /etc/foreman-proxy/settings.d/pulpcore.yml
.....UG..    /var/www/html/pub
missing     /usr/bin/pulpcore-api
missing     /usr/bin/pulpcore-content
missing     /usr/bin/pulpcore-manager
missing     /usr/bin/pulpcore-worker
S.5....T.  c /etc/foreman-installer/scenarios.d/foreman-proxy-content-answers.yaml
S.5....T.  c /etc/foreman-installer/scenarios.d/foreman-proxy-content-migrations-applied
S.5....T.  c /etc/foreman-installer/scenarios.d/foreman-proxy-content.yaml
S.5....T.  c /etc/foreman-installer/scenarios.d/katello-answers.yaml
S.5....T.  c /etc/foreman-installer/scenarios.d/katello-migrations-applied
S.5....T.  c /etc/foreman-installer/scenarios.d/katello.yaml

yes it is upgraded to 13

19

That looks pretty much the same as on my foreman server expect for those missing ones:

Those files come from python3.11-pulp-cli and python-3.11-pulpcore:

# rpm -qf /usr/bin/pulp /usr/bin/pulpcore-api /usr/bin/pulpcore-content /usr/bin/pulpcore-manager /usr/bin/pulpcore-worker 
python3.11-pulp-cli-0.29.2-2.el9.noarch
python3.11-pulpcore-3.49.22-1.el9.noarch
python3.11-pulpcore-3.49.22-1.el9.noarch
python3.11-pulpcore-3.49.22-1.el9.noarch
python3.11-pulpcore-3.49.22-1.el9.noarch

If you run the rpm command on your server it shows you the exact package names for your system, i.e. with

# rpm -qf /usr/bin/pulp /usr/bin/pulpcore-api /usr/bin/pulpcore-content /usr/bin/pulpcore-manager /usr/bin/pulpcore-worker | xargs dnf reinstall

you should be able to reinstall those two packages and it should restore the missing files. After that, foreman-installer should not complain anymore about the missing pulpcore-manager.

1 Like
20

Great .

running this was successful. now foreman-installer shows

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[OK]

Checking CA bundle size: 2
[OK]

Checking if CA bundle has trust rules: 0
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking if any Subject Alt Name on certificate matches the Subject CN
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

Checking for use of shortname as CN
[OK]

Validation succeeded


To install the Katello server with the custom certificates, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/etc/pki/tls/certs/kci_katello.pem" \
                      --certs-server-key "/etc/pki/tls/private/kci_katello.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/kci_katello_ca.pem"

To update the certificates on a currently running Katello installation, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/etc/pki/tls/certs/kci_katello.pem" \
                      --certs-server-key "/etc/pki/tls/private/kci_katello.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/kci_katello_ca.pem" \
                      --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy
2024-12-20 07:02:06 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-12-20 07:02:08 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-12-20 07:02:08 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-12-20 07:02:15 [NOTICE] [configure] Starting system configuration.
2024-12-20 07:02:26 [NOTICE] [configure] 250 configuration steps out of 1784 steps complete.
2024-12-20 07:02:27 [NOTICE] [configure] 500 configuration steps out of 1786 steps complete.
2024-12-20 07:02:30 [NOTICE] [configure] 750 configuration steps out of 1789 steps complete.
2024-12-20 07:02:30 [NOTICE] [configure] 1000 configuration steps out of 1795 steps complete.
2024-12-20 07:02:30 [NOTICE] [configure] 1250 configuration steps out of 1795 steps complete.
2024-12-20 07:02:31 [NOTICE] [configure] 1500 configuration steps out of 1795 steps complete.
2024-12-20 07:03:05 [NOTICE] [configure] 1750 configuration steps out of 1795 steps complete.
2024-12-20 07:03:09 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
Processing Repository 1/19: Rocky Linux 8 - BaseOS (1)
Processing Repository 2/19: Rocky Linux 8 - AppStream (2)
Processing Repository 3/19: EPEL-8 (3)
Processing Repository 4/19: Gitlab Runner - EL8 (7)
Processing Repository 5/19: Gitlab EE - EL8 (8)
Processing Repository 6/19: Google Compute Engine - EL8 (9)
Processing Repository 7/19: Google Cloud SDK - EL8 (10)
Processing Repository 8/19: Grafana (11)
Processing Repository 9/19: InfluxDB - EL8 (12)
Processing Repository 10/19: Remi EL8 - PHP 7.4 (15)
Processing Repository 11/19: Docker CE - EL8 (16)
Processing Repository 12/19: Jenkins (17)
Processing Repository 13/19: Rocky Linux 8 - PowerTools (18)
Processing Repository 14/19: Percona Tools - EL8 (24)
Processing Repository 15/19: Rocky Linux 8.7 - AppStream (25)
Processing Repository 16/19: Rocky Linux 9.2 - AppStream (27)
Processing Repository 17/19: Rocky Linux 9 - BaseOS (28)
Processing Repository 18/19: Rocky Linux 9 - PowerTools (29)
Processing Repository 19/19: Rocky Linux 9 - AppStream (30)
=============================================
Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
  Success!
  * Foreman is running at https://katello.kci.rocks
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar.gz"
  * Foreman Proxy is running at https://katello.kci.rocks:9090

The full log is at /var/log/foreman-installer/katello.log

I guess now only the pending issue is like

[root@katello ~]# systemctl status pulpcore-worker@2.service
â—Ź pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2024-12-20 07:06:35 UTC; 363ms ago
 Main PID: 650549 (pulpcore-worker)
    Tasks: 1 (limit: 152216)
   Memory: 29.7M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
           └─650549 /bin/python3.9 /usr/local/bin/pulpcore-worker

Dec 20 07:06:35 katello systemd[1]: Started Pulp Worker.
[root@katello ~]# systemctl status pulpcore-worker@2.service
â—Ź pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2024-12-20 07:06:37 UTC; 586ms ago
  Process: 650549 ExecStart=/usr/libexec/pulpcore/pulpcore-worker (code=exited, status=1/FAILURE)
 Main PID: 650549 (code=exited, status=1/FAILURE)

Dec 20 07:06:37 katello systemd[1]: pulpcore-worker@2.service: Main process exited, code=exited, status=1/FAILURE
Dec 20 07:06:37 katello systemd[1]: pulpcore-worker@2.service: Failed with result 'exit-code'.
[root@katello ~]# systemctl status pulpcore-worker@2.service
â—Ź pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2024-12-20 07:06:46 UTC; 437ms ago
  Process: 650574 ExecStart=/usr/libexec/pulpcore/pulpcore-worker (code=exited, status=1/FAILURE)
 Main PID: 650574 (code=exited, status=1/FAILURE)
[root@katello ~]# systemctl status pulpcore-worker@2.service
â—Ź pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2024-12-20 07:06:46 UTC; 2s ago
  Process: 650574 ExecStart=/usr/libexec/pulpcore/pulpcore-worker (code=exited, status=1/FAILURE)
 Main PID: 650574 (code=exited, status=1/FAILURE)
    Tasks: 0 (limit: 152216)
   Memory: 0B
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
[root@katello ~]# systemctl status pulpcore-worker@2.service
â—Ź pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2024-12-20 07:06:49 UTC; 1s ago
 Main PID: 650588 (pulpcore-worker)
    Tasks: 1 (limit: 152216)
   Memory: 61.9M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
           └─650588 /bin/python3.9 /usr/local/bin/pulpcore-worker

all pulpcore worker and api fails and goes active this keeps repeating does not stay active always… Once this is sorted then complete issue will be sorted