I spent a lot of time working with support to fix my install. Started over with a good install and the second I started loading products in to it I was getting failures that pulp was not working.
I ran a foremain-install --reset-data and started over again but still getting this with a foreman-maintain health check:
Check whether all services are running using the ping call: [FAIL]
Some components are failing: pulp, pulp_auth
--------------------------------------------------------------------------------
I am not seeing anything with foreman-tail so not sure what exactly is failing or why.
I did find this article Hammer ping failing or pulp and pulp_auth failing - Red Hat Customer Portal talking about how selinux is blocking things. My selinux is set to permissive but I am running the ausearch and semodule commands in /var/log/messages to clear up what I can anyway.
I cleaned up /var/log/messages grep setroubleshoot /var/log/messages | grep -o sealert.* | sort | uniq -c and ran the sealerts. But this is what I am getting now:
[root@gp2us1opsfm01-main02a ~]# journalctl -u *pulp*
Failed to add match 'my-pulpcoreapi.te': Invalid argument
[root@gp2us1opsfm01-main02a ~]# cat my-pulpcoreapi.te
module my-pulpcoreapi 1.0;
require {
type tmpfs_t;
type pulpcore_server_t;
class file { execute map read write };
}
#============= pulpcore_server_t ==============
#!!!! This avc is allowed in the current policy
allow pulpcore_server_t tmpfs_t:file { execute map read write };
Tha policy has been applied so not sure why its giving me problems. Oh and Selinux is in permissive mode and I have rebooted.
I dumped journalctl to a flat file and grepped for pulp from there. Here are some of the errors I found:
Sep 27 16:56:32 gp2us1opsfm01-main02a.example.net systemd[1]: pulpcore-api.service: Main process exited, code=dumped, status=6/ABRT
Sep 27 16:56:32 gp2us1opsfm01-main02a.example.net systemd[1]: pulpcore-api.service: Failed with result 'core-dump'.
Sep 27 16:56:32 gp2us1opsfm01-main02a.example.net systemd[1]: pulpcore-api.service: Consumed 9.471s CPU time.
Sep 27 16:56:32 gp2us1opsfm01-main02a.example.net pulpcore-content[1170]: [2024-09-27 16:56:32 +0000] [1170] [ERROR] Worker (pid:1992) was sent code 134!
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: Traceback (most recent call last):
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: return self.cursor.execute(sql)
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: ^^^^^^^^^^^^^^^^^^^^^^^^
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: raise ex.with_traceback(None)
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: This probably means the server terminated abnormally
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: before or while processing the request.
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: The above exception was the direct cause of the following exception:
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: Traceback (most recent call last):
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: File "/usr/bin/pulpcore-worker", line 33, in <module>
Sep 27 16:56:29 gp2us1opsfm01-main02a.example.net pulpcore-worker-3[1632]: sys.exit(load_entry_point('pulpcore==3.49.19', 'console_scripts', 'pulpcore-worker')())
There is quite a lot more in the logs but its way too much to upload here.
I think Selinux was not really in permissive mode at the time. I re-did the system from scratch and watched what happened when I set selinux to permissive and it did not go in to permissive. But once I got permissive mode going the re-install started working better.
