Jan 16 04:00:00 hostname systemd[1]: Starting system activity accounting tool... Jan 16 04:00:00 hostname systemd[1]: sysstat-collect.service: Succeeded. Jan 16 04:00:00 hostname systemd[1]: Started system activity accounting tool. Jan 16 04:00:01 hostname systemd[1]: Created slice User Slice of UID 994. Jan 16 04:00:01 hostname systemd[1]: Starting User runtime directory /run/user/994... Jan 16 04:00:01 hostname systemd[1]: Started User runtime directory /run/user/994. Jan 16 04:00:01 hostname systemd[1]: Starting User Manager for UID 994... Jan 16 04:00:01 hostname systemd[297161]: Reached target Paths. Jan 16 04:00:01 hostname systemd[297161]: Starting D-Bus User Message Bus Socket. Jan 16 04:00:01 hostname systemd[297161]: Reached target Timers. Jan 16 04:00:01 hostname systemd[297161]: Listening on D-Bus User Message Bus Socket. Jan 16 04:00:01 hostname systemd[297161]: Reached target Sockets. Jan 16 04:00:01 hostname systemd[297161]: Reached target Basic System. Jan 16 04:00:01 hostname systemd[297161]: Reached target Default. Jan 16 04:00:01 hostname systemd[297161]: Startup finished in 36ms. Jan 16 04:00:01 hostname systemd[1]: Started User Manager for UID 994. Jan 16 04:00:01 hostname systemd[1]: Started Session 387 of user foreman. Jan 16 04:00:04 hostname pulpcore-content[210896]: [16/Jan/2024:03:00:04 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:00:04 hostname pulpcore-content[210896]: [16/Jan/2024:03:00:04 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:00:04 hostname pulpcore-content[210896]: [16/Jan/2024:03:00:04 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:00:09 hostname systemd[1]: Stopping Foreman Proxy... Jan 16 04:00:10 hostname systemd[1]: foreman-proxy.service: Succeeded. Jan 16 04:00:10 hostname systemd[1]: Stopped Foreman Proxy. Jan 16 04:00:10 hostname systemd[1]: Stopping Foreman jobs daemon - worker-1 on sidekiq... Jan 16 04:00:10 hostname systemd[1]: Stopping Foreman jobs daemon - worker-hosts-queue-1 on sidekiq... Jan 16 04:00:12 hostname systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service: Succeeded. Jan 16 04:00:12 hostname systemd[1]: Stopped Foreman jobs daemon - worker-hosts-queue-1 on sidekiq. Jan 16 04:00:12 hostname systemd[1]: dynflow-sidekiq@worker-1.service: Succeeded. Jan 16 04:00:12 hostname systemd[1]: Stopped Foreman jobs daemon - worker-1 on sidekiq. Jan 16 04:00:12 hostname systemd[1]: Stopping Foreman... Jan 16 04:00:12 hostname foreman[210772]: [210772] === puma shutdown: 2024-01-16 04:00:12 +0100 === Jan 16 04:00:12 hostname foreman[210772]: [210772] - Goodbye! Jan 16 04:00:12 hostname foreman[210772]: [210772] - Gracefully shutting down workers... Jan 16 04:00:12 hostname systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq... Jan 16 04:00:12 hostname systemd[1]: Stopping puppetserver Service... Jan 16 04:00:12 hostname systemd[1]: Stopping The Apache HTTP Server... Jan 16 04:00:12 hostname foreman[210772]: Exiting Jan 16 04:00:12 hostname systemd[1]: foreman.service: Succeeded. Jan 16 04:00:12 hostname systemd[1]: Stopped Foreman. Jan 16 04:00:12 hostname systemd[1]: foreman.socket: Succeeded. Jan 16 04:00:12 hostname systemd[1]: Closed Foreman HTTP Server Accept Sockets. Jan 16 04:00:13 hostname systemd[1]: httpd.service: Succeeded. Jan 16 04:00:13 hostname systemd[1]: Stopped The Apache HTTP Server. Jan 16 04:00:14 hostname systemd[1]: session-387.scope: Succeeded. Jan 16 04:00:14 hostname systemd[1]: dynflow-sidekiq@orchestrator.service: Succeeded. Jan 16 04:00:14 hostname systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq. Jan 16 04:00:14 hostname systemd[1]: puppetserver.service: Succeeded. Jan 16 04:00:14 hostname systemd[1]: Stopped puppetserver Service. Jan 16 04:00:14 hostname systemd[1]: Stopping Pulp Worker... Jan 16 04:00:14 hostname systemd[1]: Stopping Pulp Worker... Jan 16 04:00:14 hostname pulpcore-worker-1[210734]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210734@hostname.server.cetin was requested to shut down gracefully. Jan 16 04:00:14 hostname systemd[1]: Stopping Pulp Worker... Jan 16 04:00:14 hostname pulpcore-worker-4[210739]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210739@hostname.server.cetin was requested to shut down gracefully. Jan 16 04:00:14 hostname systemd[1]: Stopping Pulp Worker... Jan 16 04:00:14 hostname pulpcore-worker-3[210745]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210745@hostname.server.cetin was requested to shut down gracefully. Jan 16 04:00:14 hostname systemd[1]: Stopping Apache Tomcat Web Application Container... Jan 16 04:00:14 hostname pulpcore-worker-2[210736]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210736@hostname.server.cetin was requested to shut down gracefully. Jan 16 04:00:14 hostname server[210740]: Jan 16, 2024 4:00:14 AM org.apache.coyote.AbstractProtocol pause Jan 16 04:00:14 hostname server[210740]: INFO: Pausing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"] Jan 16 04:00:14 hostname server[210740]: Jan 16, 2024 4:00:14 AM org.apache.catalina.core.StandardService stopInternal Jan 16 04:00:14 hostname server[210740]: INFO: Stopping service [Catalina] Jan 16 04:00:14 hostname pulpcore-worker-1[210734]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210734@hostname.server.cetin was shut down. Jan 16 04:00:14 hostname pulpcore-worker-4[210739]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210739@hostname.server.cetin was shut down. Jan 16 04:00:14 hostname pulpcore-worker-3[210745]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210745@hostname.server.cetin was shut down. Jan 16 04:00:14 hostname pulpcore-worker-2[210736]: pulp [None]: pulpcore.tasking.worker:INFO: Worker 210736@hostname.server.cetin was shut down. Jan 16 04:00:15 hostname systemd[1]: pulpcore-worker@1.service: Succeeded. Jan 16 04:00:15 hostname systemd[1]: Stopped Pulp Worker. Jan 16 04:00:15 hostname systemd[1]: pulpcore-worker@4.service: Succeeded. Jan 16 04:00:15 hostname systemd[1]: Stopped Pulp Worker. Jan 16 04:00:15 hostname systemd[1]: pulpcore-worker@3.service: Succeeded. Jan 16 04:00:15 hostname systemd[1]: Stopped Pulp Worker. Jan 16 04:00:15 hostname systemd[1]: pulpcore-worker@2.service: Succeeded. Jan 16 04:00:15 hostname systemd[1]: Stopped Pulp Worker. Jan 16 04:00:16 hostname server[210740]: Jan 16, 2024 4:00:16 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesObjectStreamClassCaches Jan 16 04:00:16 hostname server[210740]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.io=null" to the JVM command line arguments to enable ObjectStream cache memory leak protection. Alternatively, you can suppress this warning by disabling ObjectStream class cache memory leak protection. Jan 16 04:00:16 hostname server[210740]: Jan 16, 2024 4:00:16 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalsForLeaks Jan 16 04:00:16 hostname server[210740]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.lang=null" to the JVM command line arguments to enable ThreadLocal memory leak detection. Alternatively, you can suppress this warning by disabling ThreadLocal memory leak detection. Jan 16 04:00:16 hostname server[210740]: Jan 16, 2024 4:00:16 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesRmiTargets Jan 16 04:00:16 hostname server[210740]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.rmi/sun.rmi.transport=null" to the JVM command line arguments to enable RMI Target memory leak detection. Alternatively, you can suppress this warning by disabling RMI Target memory leak detection. Jan 16 04:00:17 hostname server[210740]: Jan 16, 2024 4:00:17 AM org.apache.coyote.AbstractProtocol stop Jan 16 04:00:17 hostname server[210740]: INFO: Stopping ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"] Jan 16 04:00:17 hostname server[210740]: Jan 16, 2024 4:00:17 AM org.apache.coyote.AbstractProtocol destroy Jan 16 04:00:17 hostname server[210740]: INFO: Destroying ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"] Jan 16 04:00:17 hostname systemd[1]: tomcat.service: Succeeded. Jan 16 04:00:17 hostname systemd[1]: Stopped Apache Tomcat Web Application Container. Jan 16 04:00:17 hostname systemd[1]: Stopping Pulp API Server... Jan 16 04:00:17 hostname systemd[1]: Stopping Pulp Content App... Jan 16 04:00:17 hostname pulpcore-content[210696]: [2024-01-16 03:00:17 +0000] [210696] [INFO] Handling signal: term Jan 16 04:00:17 hostname pulpcore-api[210770]: [2024-01-16 03:00:17 +0000] [210770] [INFO] Worker exiting (pid: 210770) Jan 16 04:00:17 hostname pulpcore-api[210760]: [2024-01-16 03:00:17 +0000] [210760] [INFO] Worker exiting (pid: 210760) Jan 16 04:00:17 hostname pulpcore-api[210742]: [2024-01-16 03:00:17 +0000] [210742] [INFO] Worker exiting (pid: 210742) Jan 16 04:00:17 hostname systemd[1]: Stopping PostgreSQL database server... Jan 16 04:00:17 hostname pulpcore-api[210746]: [2024-01-16 03:00:17 +0000] [210746] [INFO] Worker exiting (pid: 210746) Jan 16 04:00:17 hostname pulpcore-api[210722]: [2024-01-16 03:00:17 +0000] [210722] [INFO] Worker exiting (pid: 210722) Jan 16 04:00:17 hostname pulpcore-content[210720]: pulp [None]: pulpcore.content:INFO: Content App '210720@hostname.server.cetin' failed to write a heartbeat to the database, sleeping for '7' seconds. Jan 16 04:00:17 hostname pulpcore-content[210787]: pulp [None]: pulpcore.content:INFO: Content App '210787@hostname.server.cetin' failed to write a heartbeat to the database, sleeping for '7' seconds. Jan 16 04:00:17 hostname systemd[1]: Starting SSSD Kerberos Cache Manager... Jan 16 04:00:17 hostname systemd[1]: Started SSSD Kerberos Cache Manager. Jan 16 04:00:17 hostname sssd_kcm[297760]: Starting up Jan 16 04:00:17 hostname systemd[1]: postgresql.service: Killing process 210702 (postmaster) with signal SIGKILL. Jan 16 04:00:17 hostname systemd[1]: postgresql.service: Succeeded. Jan 16 04:00:17 hostname systemd[1]: Stopped PostgreSQL database server. Jan 16 04:00:18 hostname pulpcore-content[210787]: [2024-01-16 03:00:17 +0000] [210787] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210787]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210787]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210787]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210787]: return await future Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210787]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210787]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210787]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210787]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210787]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210787]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210787]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210787]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210787]: psycopg.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210787]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210787]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210787]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210787]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210787]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210787]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210787]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210787]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210787]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210787]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210787]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210787]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210787]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210787]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210787]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210787]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210787]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210787]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 198, in __enter__ Jan 16 04:00:18 hostname pulpcore-content[210787]: if not connection.get_autocommit(): Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 464, in get_autocommit Jan 16 04:00:18 hostname pulpcore-content[210787]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210787]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210787]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210787]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210787]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210787]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210787]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210787]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210787]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210787]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210787]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210787]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210787]: django.db.utils.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210787]: [2024-01-16 03:00:18 +0000] [210787] [INFO] Worker exiting (pid: 210787) Jan 16 04:00:18 hostname pulpcore-content[210720]: [2024-01-16 03:00:17 +0000] [210720] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210720]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210720]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210720]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210720]: return await future Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210720]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210720]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210720]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210720]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210720]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210720]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210720]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210720]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210720]: psycopg.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210720]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210720]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210720]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210720]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210720]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210720]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210720]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210720]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210720]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210720]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210720]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210720]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210720]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210720]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210720]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210720]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210720]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210720]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 198, in __enter__ Jan 16 04:00:18 hostname pulpcore-content[210720]: if not connection.get_autocommit(): Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 464, in get_autocommit Jan 16 04:00:18 hostname pulpcore-content[210720]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210720]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210720]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210720]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210720]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210720]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210720]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210720]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210720]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210720]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210720]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210720]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210720]: django.db.utils.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210720]: [2024-01-16 03:00:18 +0000] [210720] [INFO] Worker exiting (pid: 210720) Jan 16 04:00:18 hostname pulpcore-content[210743]: [2024-01-16 03:00:17 +0000] [210743] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210743]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210743]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210743]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210743]: return await future Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210743]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210743]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210743]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210743]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210743]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210743]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210743]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210743]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210743]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/contextlib.py", line 81, in inner Jan 16 04:00:18 hostname pulpcore-content[210743]: return func(*args, **kwds) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django_lifecycle/mixins.py", line 182, in delete Jan 16 04:00:18 hostname pulpcore-content[210743]: value = super().delete(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/base.py", line 1131, in delete Jan 16 04:00:18 hostname pulpcore-content[210743]: collector.collect([self], keep_parents=keep_parents) Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/deletion.py", line 372, in collect Jan 16 04:00:18 hostname pulpcore-content[210743]: sub_objs = field.bulk_related_objects(new_objs, self.using) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/fields.py", line 531, in bulk_related_objects Jan 16 04:00:18 hostname pulpcore-content[210743]: .get_for_model(self.model, for_concrete_model=self.for_concrete_model) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 52, in get_for_model Jan 16 04:00:18 hostname pulpcore-content[210743]: ct = self.get(app_label=opts.app_label, model=opts.model_name) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Jan 16 04:00:18 hostname pulpcore-content[210743]: return getattr(self.get_queryset(), name)(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get Jan 16 04:00:18 hostname pulpcore-content[210743]: num = len(clone) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__ Jan 16 04:00:18 hostname pulpcore-content[210743]: self._fetch_all() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all Jan 16 04:00:18 hostname pulpcore-content[210743]: self._result_cache = list(self._iterable_class(self)) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__ Jan 16 04:00:18 hostname pulpcore-content[210743]: results = compiler.execute_sql( Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql Jan 16 04:00:18 hostname pulpcore-content[210743]: cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute Jan 16 04:00:18 hostname pulpcore-content[210743]: return self._execute_with_wrappers( Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers Jan 16 04:00:18 hostname pulpcore-content[210743]: return executor(sql, params, many, context) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute Jan 16 04:00:18 hostname pulpcore-content[210743]: with self.db.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210743]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210743]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210743]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210743]: django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210743]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210743]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210743]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210743]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210743]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210743]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210743]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210743]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210743]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210743]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210743]: psycopg.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210743]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210743]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210743]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210743]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210743]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210743]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210743]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210743]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210743]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210743]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210743]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210743]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210743]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210743]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210743]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210743]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210743]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210743]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 307, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210743]: connection.set_autocommit(True) Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 483, in set_autocommit Jan 16 04:00:18 hostname pulpcore-content[210743]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210743]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210743]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210743]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210743]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210743]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210743]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210743]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210743]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210743]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210743]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210743]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210743]: django.db.utils.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210743]: [2024-01-16 03:00:18 +0000] [210743] [INFO] Worker exiting (pid: 210743) Jan 16 04:00:18 hostname pulpcore-content[210762]: [2024-01-16 03:00:17 +0000] [210762] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210762]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210762]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210762]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210762]: return await future Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210762]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210762]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210762]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210762]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210762]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210762]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210762]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210762]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210762]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/contextlib.py", line 81, in inner Jan 16 04:00:18 hostname pulpcore-content[210762]: return func(*args, **kwds) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django_lifecycle/mixins.py", line 182, in delete Jan 16 04:00:18 hostname pulpcore-content[210762]: value = super().delete(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/base.py", line 1131, in delete Jan 16 04:00:18 hostname pulpcore-content[210762]: collector.collect([self], keep_parents=keep_parents) Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/deletion.py", line 372, in collect Jan 16 04:00:18 hostname pulpcore-content[210762]: sub_objs = field.bulk_related_objects(new_objs, self.using) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/fields.py", line 531, in bulk_related_objects Jan 16 04:00:18 hostname pulpcore-content[210762]: .get_for_model(self.model, for_concrete_model=self.for_concrete_model) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 52, in get_for_model Jan 16 04:00:18 hostname pulpcore-content[210762]: ct = self.get(app_label=opts.app_label, model=opts.model_name) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Jan 16 04:00:18 hostname pulpcore-content[210762]: return getattr(self.get_queryset(), name)(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get Jan 16 04:00:18 hostname pulpcore-content[210762]: num = len(clone) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__ Jan 16 04:00:18 hostname pulpcore-content[210762]: self._fetch_all() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all Jan 16 04:00:18 hostname pulpcore-content[210762]: self._result_cache = list(self._iterable_class(self)) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__ Jan 16 04:00:18 hostname pulpcore-content[210762]: results = compiler.execute_sql( Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql Jan 16 04:00:18 hostname pulpcore-content[210762]: cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute Jan 16 04:00:18 hostname pulpcore-content[210762]: return self._execute_with_wrappers( Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers Jan 16 04:00:18 hostname pulpcore-content[210762]: return executor(sql, params, many, context) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute Jan 16 04:00:18 hostname pulpcore-content[210762]: with self.db.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210762]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210762]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210762]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210762]: django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210762]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210762]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210762]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210762]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210762]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210762]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210762]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210762]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210762]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210762]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210762]: psycopg.OperationalError: connection failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210762]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210762]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210762]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210762]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210762]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210762]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210762]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210762]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210762]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210762]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210762]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210762]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210762]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210762]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210762]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210762]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210762]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210762]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210762]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210762]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 307, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210762]: connection.set_autocommit(True) Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 483, in set_autocommit Jan 16 04:00:18 hostname pulpcore-content[210762]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210762]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210762]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210762]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210762]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210762]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210762]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210762]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210762]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210762]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210762]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210762]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210762]: django.db.utils.OperationalError: connection failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210762]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210762]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210762]: [2024-01-16 03:00:18 +0000] [210762] [INFO] Worker exiting (pid: 210762) Jan 16 04:00:18 hostname pulpcore-content[210721]: [2024-01-16 03:00:17 +0000] [210721] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210721]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210721]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210721]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210721]: return await future Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210721]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210721]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210721]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210721]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210721]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210721]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210721]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210721]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210721]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/contextlib.py", line 81, in inner Jan 16 04:00:18 hostname pulpcore-content[210721]: return func(*args, **kwds) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django_lifecycle/mixins.py", line 182, in delete Jan 16 04:00:18 hostname pulpcore-content[210721]: value = super().delete(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/base.py", line 1131, in delete Jan 16 04:00:18 hostname pulpcore-content[210721]: collector.collect([self], keep_parents=keep_parents) Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/deletion.py", line 372, in collect Jan 16 04:00:18 hostname pulpcore-content[210721]: sub_objs = field.bulk_related_objects(new_objs, self.using) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/fields.py", line 531, in bulk_related_objects Jan 16 04:00:18 hostname pulpcore-content[210721]: .get_for_model(self.model, for_concrete_model=self.for_concrete_model) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 52, in get_for_model Jan 16 04:00:18 hostname pulpcore-content[210721]: ct = self.get(app_label=opts.app_label, model=opts.model_name) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Jan 16 04:00:18 hostname pulpcore-content[210721]: return getattr(self.get_queryset(), name)(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get Jan 16 04:00:18 hostname pulpcore-content[210721]: num = len(clone) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__ Jan 16 04:00:18 hostname pulpcore-content[210721]: self._fetch_all() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all Jan 16 04:00:18 hostname pulpcore-content[210721]: self._result_cache = list(self._iterable_class(self)) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__ Jan 16 04:00:18 hostname pulpcore-content[210721]: results = compiler.execute_sql( Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql Jan 16 04:00:18 hostname pulpcore-content[210721]: cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute Jan 16 04:00:18 hostname pulpcore-content[210721]: return self._execute_with_wrappers( Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers Jan 16 04:00:18 hostname pulpcore-content[210721]: return executor(sql, params, many, context) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute Jan 16 04:00:18 hostname pulpcore-content[210721]: with self.db.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210721]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210721]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210721]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210721]: django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210721]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210721]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210721]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210721]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210721]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210721]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210721]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210721]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210721]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210721]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210721]: psycopg.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210721]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210721]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210721]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210721]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210721]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210721]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210721]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210721]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210721]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210721]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210721]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210721]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210721]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210721]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210721]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210721]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210721]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210721]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 307, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210721]: connection.set_autocommit(True) Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 483, in set_autocommit Jan 16 04:00:18 hostname pulpcore-content[210721]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210721]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210721]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210721]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210721]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210721]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210721]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210721]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210721]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210721]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210721]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210721]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210721]: django.db.utils.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210721]: [2024-01-16 03:00:18 +0000] [210721] [INFO] Worker exiting (pid: 210721) Jan 16 04:00:18 hostname pulpcore-content[210896]: [2024-01-16 03:00:17 +0000] [210896] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210896]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210896]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210896]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210896]: return await future Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210896]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210896]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210896]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210896]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210896]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210896]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210896]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210896]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210896]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/contextlib.py", line 81, in inner Jan 16 04:00:18 hostname pulpcore-content[210896]: return func(*args, **kwds) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django_lifecycle/mixins.py", line 182, in delete Jan 16 04:00:18 hostname pulpcore-content[210896]: value = super().delete(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/base.py", line 1131, in delete Jan 16 04:00:18 hostname pulpcore-content[210896]: collector.collect([self], keep_parents=keep_parents) Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/deletion.py", line 372, in collect Jan 16 04:00:18 hostname pulpcore-content[210896]: sub_objs = field.bulk_related_objects(new_objs, self.using) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/fields.py", line 531, in bulk_related_objects Jan 16 04:00:18 hostname pulpcore-content[210896]: .get_for_model(self.model, for_concrete_model=self.for_concrete_model) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 52, in get_for_model Jan 16 04:00:18 hostname pulpcore-content[210896]: ct = self.get(app_label=opts.app_label, model=opts.model_name) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Jan 16 04:00:18 hostname pulpcore-content[210896]: return getattr(self.get_queryset(), name)(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get Jan 16 04:00:18 hostname pulpcore-content[210896]: num = len(clone) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__ Jan 16 04:00:18 hostname pulpcore-content[210896]: self._fetch_all() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all Jan 16 04:00:18 hostname pulpcore-content[210896]: self._result_cache = list(self._iterable_class(self)) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__ Jan 16 04:00:18 hostname pulpcore-content[210896]: results = compiler.execute_sql( Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql Jan 16 04:00:18 hostname pulpcore-content[210896]: cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute Jan 16 04:00:18 hostname pulpcore-content[210896]: return self._execute_with_wrappers( Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers Jan 16 04:00:18 hostname pulpcore-content[210896]: return executor(sql, params, many, context) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute Jan 16 04:00:18 hostname pulpcore-content[210896]: with self.db.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210896]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210896]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210896]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210896]: django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210896]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210896]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210896]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210896]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210896]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210896]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210896]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210896]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210896]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210896]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210896]: psycopg.OperationalError: connection failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210896]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210896]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210896]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210896]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210896]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210896]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210896]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210896]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210896]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210896]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210896]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210896]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210896]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210896]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210896]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210896]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210896]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210896]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210896]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210896]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 307, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210896]: connection.set_autocommit(True) Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 483, in set_autocommit Jan 16 04:00:18 hostname pulpcore-content[210896]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210896]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210896]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210896]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210896]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210896]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210896]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210896]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210896]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210896]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210896]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210896]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210896]: django.db.utils.OperationalError: connection failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210896]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210896]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210896]: [2024-01-16 03:00:18 +0000] [210896] [INFO] Worker exiting (pid: 210896) Jan 16 04:00:18 hostname pulpcore-content[210908]: [2024-01-16 03:00:17 +0000] [210908] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210908]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210908]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210908]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210908]: return await future Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210908]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210908]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210908]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210908]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210908]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210908]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210908]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210908]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210908]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/contextlib.py", line 81, in inner Jan 16 04:00:18 hostname pulpcore-content[210908]: return func(*args, **kwds) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django_lifecycle/mixins.py", line 182, in delete Jan 16 04:00:18 hostname pulpcore-content[210908]: value = super().delete(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/base.py", line 1131, in delete Jan 16 04:00:18 hostname pulpcore-content[210908]: collector.collect([self], keep_parents=keep_parents) Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/deletion.py", line 372, in collect Jan 16 04:00:18 hostname pulpcore-content[210908]: sub_objs = field.bulk_related_objects(new_objs, self.using) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/fields.py", line 531, in bulk_related_objects Jan 16 04:00:18 hostname pulpcore-content[210908]: .get_for_model(self.model, for_concrete_model=self.for_concrete_model) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 52, in get_for_model Jan 16 04:00:18 hostname pulpcore-content[210908]: ct = self.get(app_label=opts.app_label, model=opts.model_name) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Jan 16 04:00:18 hostname pulpcore-content[210908]: return getattr(self.get_queryset(), name)(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get Jan 16 04:00:18 hostname pulpcore-content[210908]: num = len(clone) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__ Jan 16 04:00:18 hostname pulpcore-content[210908]: self._fetch_all() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all Jan 16 04:00:18 hostname pulpcore-content[210908]: self._result_cache = list(self._iterable_class(self)) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__ Jan 16 04:00:18 hostname pulpcore-content[210908]: results = compiler.execute_sql( Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql Jan 16 04:00:18 hostname pulpcore-content[210908]: cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute Jan 16 04:00:18 hostname pulpcore-content[210908]: return self._execute_with_wrappers( Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers Jan 16 04:00:18 hostname pulpcore-content[210908]: return executor(sql, params, many, context) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute Jan 16 04:00:18 hostname pulpcore-content[210908]: with self.db.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210908]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210908]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210908]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210908]: django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210908]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210908]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210908]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210908]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210908]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210908]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210908]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210908]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210908]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210908]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210908]: psycopg.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210908]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210908]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210908]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210908]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210908]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210908]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210908]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210908]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210908]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210908]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210908]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210908]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210908]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210908]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210908]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210908]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210908]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210908]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 307, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210908]: connection.set_autocommit(True) Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 483, in set_autocommit Jan 16 04:00:18 hostname pulpcore-content[210908]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210908]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210908]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210908]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210908]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210908]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210908]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210908]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210908]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210908]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210908]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210908]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210908]: django.db.utils.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210731]: [2024-01-16 03:00:17 +0000] [210731] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210731]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210731]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210731]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210731]: return await future Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210731]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210731]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210731]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210731]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210731]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210731]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210731]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210731]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210731]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/contextlib.py", line 81, in inner Jan 16 04:00:18 hostname pulpcore-content[210731]: return func(*args, **kwds) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django_lifecycle/mixins.py", line 182, in delete Jan 16 04:00:18 hostname pulpcore-content[210731]: value = super().delete(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/base.py", line 1131, in delete Jan 16 04:00:18 hostname pulpcore-content[210731]: collector.collect([self], keep_parents=keep_parents) Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/deletion.py", line 372, in collect Jan 16 04:00:18 hostname pulpcore-content[210731]: sub_objs = field.bulk_related_objects(new_objs, self.using) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/fields.py", line 531, in bulk_related_objects Jan 16 04:00:18 hostname pulpcore-content[210731]: .get_for_model(self.model, for_concrete_model=self.for_concrete_model) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 52, in get_for_model Jan 16 04:00:18 hostname pulpcore-content[210731]: ct = self.get(app_label=opts.app_label, model=opts.model_name) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Jan 16 04:00:18 hostname pulpcore-content[210731]: return getattr(self.get_queryset(), name)(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get Jan 16 04:00:18 hostname pulpcore-content[210731]: num = len(clone) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__ Jan 16 04:00:18 hostname pulpcore-content[210731]: self._fetch_all() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all Jan 16 04:00:18 hostname pulpcore-content[210731]: self._result_cache = list(self._iterable_class(self)) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__ Jan 16 04:00:18 hostname pulpcore-content[210731]: results = compiler.execute_sql( Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql Jan 16 04:00:18 hostname pulpcore-content[210731]: cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute Jan 16 04:00:18 hostname pulpcore-content[210731]: return self._execute_with_wrappers( Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers Jan 16 04:00:18 hostname pulpcore-content[210731]: return executor(sql, params, many, context) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute Jan 16 04:00:18 hostname pulpcore-content[210731]: with self.db.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210731]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210731]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210731]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210731]: django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210731]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210731]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210731]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210731]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210731]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210731]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210731]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210731]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210731]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210731]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210731]: psycopg.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210731]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210731]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210731]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210731]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210731]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210731]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210731]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210731]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210731]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210731]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210731]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210731]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210731]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210731]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210731]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210731]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210731]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210731]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 307, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210731]: connection.set_autocommit(True) Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 483, in set_autocommit Jan 16 04:00:18 hostname pulpcore-content[210731]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210731]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210731]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210731]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210731]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210731]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210731]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210731]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210731]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210731]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210731]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210731]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210731]: django.db.utils.OperationalError: connection failed: the database system is shutting down Jan 16 04:00:18 hostname pulpcore-content[210731]: [2024-01-16 03:00:18 +0000] [210731] [INFO] Worker exiting (pid: 210731) Jan 16 04:00:18 hostname pulpcore-content[210923]: [2024-01-16 03:00:17 +0000] [210923] [ERROR] Exception in gunicorn worker Jan 16 04:00:18 hostname pulpcore-content[210923]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210923]: raise exc_info[1] Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 68, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210923]: await asyncio.sleep(heartbeat_interval) Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 639, in sleep Jan 16 04:00:18 hostname pulpcore-content[210923]: return await future Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: asyncio.exceptions.CancelledError Jan 16 04:00:18 hostname pulpcore-content[210923]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210923]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210923]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210923]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210923]: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210923]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210923]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210923]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210923]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/contextlib.py", line 81, in inner Jan 16 04:00:18 hostname pulpcore-content[210923]: return func(*args, **kwds) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django_lifecycle/mixins.py", line 182, in delete Jan 16 04:00:18 hostname pulpcore-content[210923]: value = super().delete(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/base.py", line 1131, in delete Jan 16 04:00:18 hostname pulpcore-content[210923]: collector.collect([self], keep_parents=keep_parents) Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/deletion.py", line 372, in collect Jan 16 04:00:18 hostname pulpcore-content[210923]: sub_objs = field.bulk_related_objects(new_objs, self.using) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/fields.py", line 531, in bulk_related_objects Jan 16 04:00:18 hostname pulpcore-content[210923]: .get_for_model(self.model, for_concrete_model=self.for_concrete_model) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 52, in get_for_model Jan 16 04:00:18 hostname pulpcore-content[210923]: ct = self.get(app_label=opts.app_label, model=opts.model_name) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Jan 16 04:00:18 hostname pulpcore-content[210923]: return getattr(self.get_queryset(), name)(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get Jan 16 04:00:18 hostname pulpcore-content[210923]: num = len(clone) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__ Jan 16 04:00:18 hostname pulpcore-content[210923]: self._fetch_all() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all Jan 16 04:00:18 hostname pulpcore-content[210923]: self._result_cache = list(self._iterable_class(self)) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__ Jan 16 04:00:18 hostname pulpcore-content[210923]: results = compiler.execute_sql( Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql Jan 16 04:00:18 hostname pulpcore-content[210923]: cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute Jan 16 04:00:18 hostname pulpcore-content[210923]: return self._execute_with_wrappers( Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers Jan 16 04:00:18 hostname pulpcore-content[210923]: return executor(sql, params, many, context) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute Jan 16 04:00:18 hostname pulpcore-content[210923]: with self.db.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210923]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute Jan 16 04:00:18 hostname pulpcore-content[210923]: return self.cursor.execute(sql, params) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute Jan 16 04:00:18 hostname pulpcore-content[210923]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210923]: django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210923]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210923]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210923]: During handling of the above exception, another exception occurred: Jan 16 04:00:18 hostname pulpcore-content[210923]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210923]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210923]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210923]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210923]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210923]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210923]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210923]: psycopg.OperationalError: connection failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210923]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210923]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210923]: The above exception was the direct cause of the following exception: Jan 16 04:00:18 hostname pulpcore-content[210923]: Traceback (most recent call last): Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 57, in run Jan 16 04:00:18 hostname pulpcore-content[210923]: self.loop.run_until_complete(self._task) Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete Jan 16 04:00:18 hostname pulpcore-content[210923]: return future.result() Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/site-packages/aiohttp/worker.py", line 131, in _run Jan 16 04:00:18 hostname pulpcore-content[210923]: await runner.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 294, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210923]: await self._cleanup_server() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_runner.py", line 381, in _cleanup_server Jan 16 04:00:18 hostname pulpcore-content[210923]: await self._app.cleanup() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 432, in cleanup Jan 16 04:00:18 hostname pulpcore-content[210923]: await self.on_cleanup.send(self) Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/aiosignal/__init__.py", line 36, in send Jan 16 04:00:18 hostname pulpcore-content[210923]: await receiver(*args, **kwargs) # type: ignore Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 555, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210923]: raise errors[0] Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 546, in _on_cleanup Jan 16 04:00:18 hostname pulpcore-content[210923]: await it.__anext__() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 79, in _heartbeat_ctx Jan 16 04:00:18 hostname pulpcore-content[210923]: await heartbeat_task Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/pulpcore/content/__init__.py", line 71, in _heartbeat Jan 16 04:00:18 hostname pulpcore-content[210923]: await content_app_status.adelete() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ Jan 16 04:00:18 hostname pulpcore-content[210923]: ret = await asyncio.wait_for(future, timeout=None) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 442, in wait_for Jan 16 04:00:18 hostname pulpcore-content[210923]: return await fut Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run Jan 16 04:00:18 hostname pulpcore-content[210923]: result = self.fn(*self.args, **self.kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 488, in thread_handler Jan 16 04:00:18 hostname pulpcore-content[210923]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib64/python3.11/contextlib.py", line 80, in inner Jan 16 04:00:18 hostname pulpcore-content[210923]: with self._recreate_cm(): Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/transaction.py", line 307, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210923]: connection.set_autocommit(True) Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 483, in set_autocommit Jan 16 04:00:18 hostname pulpcore-content[210923]: self.ensure_connection() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210923]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210923]: with self.wrap_database_errors: Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ Jan 16 04:00:18 hostname pulpcore-content[210923]: raise dj_exc_value.with_traceback(traceback) from exc_value Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection Jan 16 04:00:18 hostname pulpcore-content[210923]: self.connect() Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210923]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect Jan 16 04:00:18 hostname pulpcore-content[210923]: self.connection = self.get_new_connection(conn_params) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner Jan 16 04:00:18 hostname pulpcore-content[210923]: return func(*args, **kwargs) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection Jan 16 04:00:18 hostname pulpcore-content[210923]: connection = self.Database.connect(**conn_params) Jan 16 04:00:18 hostname pulpcore-content[210923]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 16 04:00:18 hostname pulpcore-content[210923]: File "/usr/lib/python3.11/site-packages/psycopg/connection.py", line 728, in connect Jan 16 04:00:18 hostname pulpcore-content[210923]: raise ex.with_traceback(None) Jan 16 04:00:18 hostname pulpcore-content[210923]: django.db.utils.OperationalError: connection failed: server closed the connection unexpectedly Jan 16 04:00:18 hostname pulpcore-content[210923]: #011This probably means the server terminated abnormally Jan 16 04:00:18 hostname pulpcore-content[210923]: #011before or while processing the request. Jan 16 04:00:18 hostname pulpcore-content[210923]: [2024-01-16 03:00:18 +0000] [210923] [INFO] Worker exiting (pid: 210923) Jan 16 04:00:18 hostname pulpcore-content[210908]: [2024-01-16 03:00:18 +0000] [210908] [INFO] Worker exiting (pid: 210908) Jan 16 04:00:18 hostname systemd[1]: Started Process Core Dump (PID 297764/UID 0). Jan 16 04:00:18 hostname systemd[1]: Started Process Core Dump (PID 297765/UID 0). Jan 16 04:00:18 hostname systemd[1]: Started Process Core Dump (PID 297763/UID 0). Jan 16 04:00:18 hostname systemd[1]: Started Process Core Dump (PID 297766/UID 0). Jan 16 04:00:19 hostname systemd-coredump[297768]: Resource limits disable core dumping for process 210742 (pulpcore-api). Jan 16 04:00:19 hostname systemd-coredump[297768]: Process 210742 (pulpcore-api) of user 991 dumped core. Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@43-297765-0.service: Succeeded. Jan 16 04:00:19 hostname systemd-coredump[297770]: Resource limits disable core dumping for process 210746 (pulpcore-api). Jan 16 04:00:19 hostname systemd-coredump[297770]: Process 210746 (pulpcore-api) of user 991 dumped core. Jan 16 04:00:19 hostname systemd-coredump[297769]: Resource limits disable core dumping for process 210722 (pulpcore-api). Jan 16 04:00:19 hostname systemd-coredump[297769]: Process 210722 (pulpcore-api) of user 991 dumped core. Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@42-297763-0.service: Succeeded. Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@45-297766-0.service: Succeeded. Jan 16 04:00:19 hostname systemd-coredump[297767]: Resource limits disable core dumping for process 210770 (pulpcore-api). Jan 16 04:00:19 hostname systemd-coredump[297767]: Process 210770 (pulpcore-api) of user 991 dumped core. Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@44-297764-0.service: Succeeded. Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297793/UID 0). Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297787/UID 0). Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297796/UID 0). Jan 16 04:00:19 hostname systemd-coredump[297794]: Resource limits disable core dumping for process 210720 (pulpcore-conten). Jan 16 04:00:19 hostname systemd-coredump[297794]: Process 210720 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@47-297793-0.service: Succeeded. Jan 16 04:00:19 hostname systemd-coredump[297795]: Resource limits disable core dumping for process 210787 (pulpcore-conten). Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210720 was terminated due to signal 6 Jan 16 04:00:19 hostname systemd-coredump[297795]: Process 210787 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210787 was terminated due to signal 6 Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@46-297787-0.service: Succeeded. Jan 16 04:00:19 hostname systemd-coredump[297797]: Resource limits disable core dumping for process 210762 (pulpcore-conten). Jan 16 04:00:19 hostname systemd-coredump[297797]: Process 210762 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210762 was terminated due to signal 6 Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@48-297796-0.service: Succeeded. Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297814/UID 0). Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297816/UID 0). Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297818/UID 0). Jan 16 04:00:19 hostname pulpcore-api[210698]: [2024-01-16 03:00:19 +0000] [210698] [WARNING] Worker with pid 210770 was terminated due to signal 6 Jan 16 04:00:19 hostname pulpcore-api[210698]: [2024-01-16 03:00:19 +0000] [210698] [WARNING] Worker with pid 210746 was terminated due to signal 6 Jan 16 04:00:19 hostname pulpcore-api[210698]: [2024-01-16 03:00:19 +0000] [210698] [WARNING] Worker with pid 210722 was terminated due to signal 6 Jan 16 04:00:19 hostname pulpcore-api[210698]: [2024-01-16 03:00:19 +0000] [210698] [WARNING] Worker with pid 210742 was terminated due to signal 6 Jan 16 04:00:19 hostname pulpcore-api[210698]: [2024-01-16 03:00:18 +0000] [210698] [INFO] Handling signal: term Jan 16 04:00:19 hostname systemd-coredump[297815]: Resource limits disable core dumping for process 210908 (pulpcore-conten). Jan 16 04:00:19 hostname systemd-coredump[297815]: Process 210908 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210908 was terminated due to signal 6 Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@49-297814-0.service: Succeeded. Jan 16 04:00:19 hostname systemd-coredump[297817]: Resource limits disable core dumping for process 210896 (pulpcore-conten). Jan 16 04:00:19 hostname systemd-coredump[297817]: Process 210896 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297825/UID 0). Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210896 was terminated due to signal 6 Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@50-297816-0.service: Succeeded. Jan 16 04:00:19 hostname systemd[1]: Started Process Core Dump (PID 297832/UID 0). Jan 16 04:00:19 hostname pulpcore-api[210698]: [2024-01-16 03:00:19 +0000] [210698] [INFO] Shutting down: Master Jan 16 04:00:19 hostname systemd-coredump[297819]: Resource limits disable core dumping for process 210923 (pulpcore-conten). Jan 16 04:00:19 hostname systemd-coredump[297819]: Process 210923 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210923 was terminated due to signal 6 Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@51-297818-0.service: Succeeded. Jan 16 04:00:19 hostname systemd-coredump[297826]: Resource limits disable core dumping for process 210731 (pulpcore-conten). Jan 16 04:00:19 hostname systemd-coredump[297826]: Process 210731 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210731 was terminated due to signal 6 Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@52-297825-0.service: Succeeded. Jan 16 04:00:19 hostname systemd-coredump[297833]: Resource limits disable core dumping for process 210743 (pulpcore-conten). Jan 16 04:00:19 hostname systemd-coredump[297833]: Process 210743 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:19 hostname systemd[1]: systemd-coredump@53-297832-0.service: Succeeded. Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [WARNING] Worker with pid 210743 was terminated due to signal 6 Jan 16 04:00:19 hostname pulpcore-content[210696]: [2024-01-16 03:00:19 +0000] [210696] [INFO] Shutting down: Master Jan 16 04:00:20 hostname systemd[1]: Started Process Core Dump (PID 297850/UID 0). Jan 16 04:00:20 hostname systemd-coredump[297851]: Resource limits disable core dumping for process 210696 (pulpcore-conten). Jan 16 04:00:20 hostname systemd-coredump[297851]: Process 210696 (pulpcore-conten) of user 991 dumped core. Jan 16 04:00:20 hostname systemd[1]: pulpcore-content.service: Main process exited, code=dumped, status=6/ABRT Jan 16 04:00:20 hostname systemd[1]: pulpcore-content.service: Failed with result 'core-dump'. Jan 16 04:00:20 hostname systemd[1]: Stopped Pulp Content App. Jan 16 04:00:20 hostname systemd[1]: pulpcore-content.socket: Succeeded. Jan 16 04:00:20 hostname systemd[1]: Closed Pulp Content App socket. Jan 16 04:00:20 hostname systemd[1]: systemd-coredump@54-297850-0.service: Succeeded. Jan 16 04:00:20 hostname systemd[1]: Started Process Core Dump (PID 297859/UID 0). Jan 16 04:00:21 hostname systemd-coredump[297860]: Resource limits disable core dumping for process 210698 (pulpcore-api). Jan 16 04:00:21 hostname systemd-coredump[297860]: Process 210698 (pulpcore-api) of user 991 dumped core. Jan 16 04:00:21 hostname systemd[1]: systemd-coredump@55-297859-0.service: Succeeded. Jan 16 04:00:21 hostname systemd[1]: pulpcore-api.service: Main process exited, code=dumped, status=6/ABRT Jan 16 04:00:21 hostname systemd[1]: pulpcore-api.service: Failed with result 'core-dump'. Jan 16 04:00:21 hostname systemd[1]: Stopped Pulp API Server. Jan 16 04:00:21 hostname systemd[1]: pulpcore-api.socket: Succeeded. Jan 16 04:00:21 hostname systemd[1]: Closed Pulp API Server socket. Jan 16 04:00:21 hostname systemd[1]: Stopping Redis persistent key-value database... Jan 16 04:00:21 hostname systemd[1]: redis.service: Succeeded. Jan 16 04:00:21 hostname systemd[1]: Stopped Redis persistent key-value database. Jan 16 04:00:21 hostname systemd[1]: Started Device-mapper event daemon. Jan 16 04:00:21 hostname dmeventd[297927]: dmeventd ready for processing. Jan 16 04:00:21 hostname dmeventd[297927]: Monitoring snapshot vg_katello-lv_pgsql_snapshot. Jan 16 04:00:24 hostname systemd[1]: Stopping User Manager for UID 994... Jan 16 04:00:24 hostname systemd[297161]: Stopped target Default. Jan 16 04:00:24 hostname systemd[297161]: Stopped target Basic System. Jan 16 04:00:24 hostname systemd[297161]: Stopped target Sockets. Jan 16 04:00:24 hostname systemd[297161]: Closed D-Bus User Message Bus Socket. Jan 16 04:00:24 hostname systemd[297161]: Stopped target Paths. Jan 16 04:00:24 hostname systemd[297161]: Stopped target Timers. Jan 16 04:00:24 hostname systemd[297161]: Reached target Shutdown. Jan 16 04:00:24 hostname systemd[297161]: Started Exit the Session. Jan 16 04:00:24 hostname systemd[297161]: Reached target Exit the Session. Jan 16 04:00:24 hostname systemd[1]: user@994.service: Succeeded. Jan 16 04:00:24 hostname systemd[1]: Stopped User Manager for UID 994. Jan 16 04:00:24 hostname systemd[1]: Stopping User runtime directory /run/user/994... Jan 16 04:00:24 hostname systemd[1]: run-user-994.mount: Succeeded. Jan 16 04:00:24 hostname systemd[1]: user-runtime-dir@994.service: Succeeded. Jan 16 04:00:24 hostname systemd[1]: Stopped User runtime directory /run/user/994. Jan 16 04:00:24 hostname systemd[1]: Removed slice User Slice of UID 994. Jan 16 04:00:25 hostname kernel: XFS (dm-10): Mounting V5 Filesystem Jan 16 04:00:25 hostname kernel: XFS (dm-10): Ending clean mount Jan 16 04:00:26 hostname dmeventd[297927]: Monitoring snapshot vg_katello-lv_pulp_snapshot. Jan 16 04:01:07 hostname kernel: XFS (dm-13): Mounting V5 Filesystem Jan 16 04:01:07 hostname kernel: XFS (dm-13): Ending clean mount Jan 16 04:01:09 hostname systemd[1]: Starting Redis persistent key-value database... Jan 16 04:01:09 hostname systemd[1]: Started Redis persistent key-value database. Jan 16 04:01:09 hostname systemd[1]: Starting Pulp Content App socket. Jan 16 04:01:09 hostname systemd[1]: Starting Pulp API Server socket. Jan 16 04:01:09 hostname systemd[1]: Listening on Pulp Content App socket. Jan 16 04:01:09 hostname systemd[1]: Starting Pulp Content App... Jan 16 04:01:09 hostname systemd[1]: Listening on Pulp API Server socket. Jan 16 04:01:09 hostname systemd[1]: Starting Pulp API Server... Jan 16 04:01:09 hostname systemd[1]: Starting PostgreSQL database server... Jan 16 04:01:09 hostname postmaster[298813]: 2024-01-16 04:01:09 CET LOG: starting PostgreSQL 12.17 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit Jan 16 04:01:09 hostname postmaster[298813]: 2024-01-16 04:01:09 CET LOG: listening on IPv6 address "::1", port 5432 Jan 16 04:01:09 hostname postmaster[298813]: 2024-01-16 04:01:09 CET LOG: listening on IPv4 address "127.0.0.1", port 5432 Jan 16 04:01:09 hostname postmaster[298813]: 2024-01-16 04:01:09 CET LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" Jan 16 04:01:09 hostname postmaster[298813]: 2024-01-16 04:01:09 CET LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" Jan 16 04:01:09 hostname postmaster[298813]: 2024-01-16 04:01:09 CET LOG: redirecting log output to logging collector process Jan 16 04:01:09 hostname postmaster[298813]: 2024-01-16 04:01:09 CET HINT: Future log output will appear in directory "log". Jan 16 04:01:09 hostname systemd[1]: Started PostgreSQL database server. Jan 16 04:01:24 hostname pulpcore-content[298808]: [2024-01-16 03:01:24 +0000] [298808] [INFO] Starting gunicorn 20.1.0 Jan 16 04:01:24 hostname pulpcore-content[298808]: [2024-01-16 03:01:24 +0000] [298808] [INFO] Listening at: unix:/run/pulpcore-content.sock (298808) Jan 16 04:01:24 hostname pulpcore-content[298808]: [2024-01-16 03:01:24 +0000] [298808] [INFO] Using worker: aiohttp.GunicornWebWorker Jan 16 04:01:24 hostname systemd[1]: Started Pulp Content App. Jan 16 04:01:24 hostname pulpcore-content[298833]: [2024-01-16 03:01:24 +0000] [298833] [INFO] Booting worker with pid: 298833 Jan 16 04:01:24 hostname pulpcore-content[298834]: [2024-01-16 03:01:24 +0000] [298834] [INFO] Booting worker with pid: 298834 Jan 16 04:01:24 hostname pulpcore-content[298838]: [2024-01-16 03:01:24 +0000] [298838] [INFO] Booting worker with pid: 298838 Jan 16 04:01:24 hostname pulpcore-api[298809]: [2024-01-16 03:01:24 +0000] [298809] [INFO] Starting gunicorn 20.1.0 Jan 16 04:01:24 hostname pulpcore-api[298809]: [2024-01-16 03:01:24 +0000] [298809] [INFO] Listening at: unix:/run/pulpcore-api.sock (298809) Jan 16 04:01:24 hostname pulpcore-api[298809]: [2024-01-16 03:01:24 +0000] [298809] [INFO] Using worker: pulpcore.app.entrypoint.PulpApiWorker Jan 16 04:01:24 hostname systemd[1]: Started Pulp API Server. Jan 16 04:01:24 hostname pulpcore-api[298840]: [2024-01-16 03:01:24 +0000] [298840] [INFO] Booting worker with pid: 298840 Jan 16 04:01:24 hostname pulpcore-api[298847]: [2024-01-16 03:01:24 +0000] [298847] [INFO] Booting worker with pid: 298847 Jan 16 04:01:24 hostname systemd[1]: Started Pulp Worker. Jan 16 04:01:24 hostname systemd[1]: Started Pulp Worker. Jan 16 04:01:24 hostname systemd[1]: Started Pulp Worker. Jan 16 04:01:24 hostname pulpcore-content[298861]: [2024-01-16 03:01:24 +0000] [298861] [INFO] Booting worker with pid: 298861 Jan 16 04:01:24 hostname systemd[1]: Started Pulp Worker. Jan 16 04:01:24 hostname pulpcore-api[298864]: [2024-01-16 03:01:24 +0000] [298864] [INFO] Booting worker with pid: 298864 Jan 16 04:01:24 hostname systemd[1]: Started Apache Tomcat Web Application Container. Jan 16 04:01:24 hostname pulpcore-api[298866]: [2024-01-16 03:01:24 +0000] [298866] [INFO] Booting worker with pid: 298866 Jan 16 04:01:24 hostname pulpcore-content[298868]: [2024-01-16 03:01:24 +0000] [298868] [INFO] Booting worker with pid: 298868 Jan 16 04:01:24 hostname server[298867]: Java virtual machine used: /usr/lib/jvm/jre-17/bin/java Jan 16 04:01:24 hostname server[298867]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar Jan 16 04:01:24 hostname server[298867]: main class used: org.apache.catalina.startup.Bootstrap Jan 16 04:01:24 hostname server[298867]: flags used: -Xms1024m -Xmx4096m -Dcom.redhat.fips=false -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config Jan 16 04:01:24 hostname server[298867]: options used: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Jan 16 04:01:24 hostname server[298867]: arguments used: start Jan 16 04:01:24 hostname systemd[1]: Starting puppetserver Service... Jan 16 04:01:24 hostname systemd[1]: Starting Foreman HTTP Server Accept Sockets. Jan 16 04:01:24 hostname pulpcore-api[298889]: [2024-01-16 03:01:24 +0000] [298889] [INFO] Booting worker with pid: 298889 Jan 16 04:01:24 hostname systemd[1]: Listening on Foreman HTTP Server Accept Sockets. Jan 16 04:01:24 hostname systemd[1]: Starting Foreman... Jan 16 04:01:24 hostname systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq... Jan 16 04:01:24 hostname systemd[1]: Starting The Apache HTTP Server... Jan 16 04:01:24 hostname pulpcore-content[298883]: [2024-01-16 03:01:24 +0000] [298883] [INFO] Booting worker with pid: 298883 Jan 16 04:01:24 hostname pulpcore-content[298904]: [2024-01-16 03:01:24 +0000] [298904] [INFO] Booting worker with pid: 298904 Jan 16 04:01:24 hostname pulpcore-content[298909]: [2024-01-16 03:01:24 +0000] [298909] [INFO] Booting worker with pid: 298909 Jan 16 04:01:25 hostname pulpcore-content[298924]: [2024-01-16 03:01:25 +0000] [298924] [INFO] Booting worker with pid: 298924 Jan 16 04:01:25 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:01:25.266Z pid=298895 tid=6dqr INFO: Enabling systemd notification integration Jan 16 04:01:25 hostname systemd[1]: Started The Apache HTTP Server. Jan 16 04:01:25 hostname httpd[298896]: Server configured, listening on: port 80, port 443 Jan 16 04:01:26 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:01:26.366Z pid=298895 tid=6dqr INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"} Jan 16 04:01:26 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:01:26.368Z pid=298895 tid=6dqr INFO: GitLab reliable fetch activated! Jan 16 04:01:26 hostname server[298867]: Jan 16, 2024 4:01:26 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent Jan 16 04:01:26 hostname server[298867]: INFO: The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib] Jan 16 04:01:27 hostname server[298867]: Jan 16, 2024 4:01:27 AM org.apache.coyote.AbstractProtocol init Jan 16 04:01:27 hostname server[298867]: INFO: Initializing ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"] Jan 16 04:01:28 hostname foreman[298891]: => Booting Puma Jan 16 04:01:28 hostname foreman[298891]: => Rails 6.1.7.6 application starting in production Jan 16 04:01:28 hostname foreman[298891]: => Run `bin/rails server --help` for more startup options Jan 16 04:01:30 hostname server[298867]: Jan 16, 2024 4:01:30 AM org.apache.catalina.startup.Catalina load Jan 16 04:01:30 hostname server[298867]: INFO: Server initialization in [4729] milliseconds Jan 16 04:01:30 hostname server[298867]: Jan 16, 2024 4:01:30 AM org.apache.catalina.core.StandardService startInternal Jan 16 04:01:30 hostname server[298867]: INFO: Starting service [Catalina] Jan 16 04:01:30 hostname server[298867]: Jan 16, 2024 4:01:30 AM org.apache.catalina.core.StandardEngine startInternal Jan 16 04:01:30 hostname server[298867]: INFO: Starting Servlet engine: [Apache Tomcat/9.0.50] Jan 16 04:01:30 hostname server[298867]: Jan 16, 2024 4:01:30 AM org.apache.catalina.startup.HostConfig deployDirectory Jan 16 04:01:30 hostname server[298867]: INFO: Deploying web application directory [/var/lib/tomcat/webapps/candlepin] Jan 16 04:01:30 hostname pulpcore-worker-3[298856]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker Jan 16 04:01:31 hostname pulpcore-worker-3[298856]: pulp [None]: pulpcore.tasking.worker:INFO: New worker '298856@hostname.server.cetin' discovered Jan 16 04:01:32 hostname pulpcore-worker-4[298862]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker Jan 16 04:01:32 hostname pulpcore-worker-1[298852]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker Jan 16 04:01:32 hostname pulpcore-worker-4[298862]: pulp [None]: pulpcore.tasking.worker:INFO: New worker '298862@hostname.server.cetin' discovered Jan 16 04:01:32 hostname pulpcore-worker-2[298858]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker Jan 16 04:01:32 hostname pulpcore-worker-1[298852]: pulp [None]: pulpcore.tasking.worker:INFO: New worker '298852@hostname.server.cetin' discovered Jan 16 04:01:32 hostname pulpcore-worker-2[298858]: pulp [None]: pulpcore.tasking.worker:INFO: New worker '298858@hostname.server.cetin' discovered Jan 16 04:01:38 hostname puppetserver[298941]: WARNING: abs already refers to: #'clojure.core/abs in namespace: medley.core, being replaced by: #'medley.core/abs Jan 16 04:01:42 hostname foreman[298891]: [298891] * Enabling systemd notification integration Jan 16 04:01:42 hostname foreman[298891]: [298891] Puma starting in cluster mode... Jan 16 04:01:42 hostname foreman[298891]: [298891] * Puma version: 6.4.0 (ruby 2.7.8-p225) ("The Eagle of Durango") Jan 16 04:01:42 hostname foreman[298891]: [298891] * Min threads: 5 Jan 16 04:01:42 hostname foreman[298891]: [298891] * Max threads: 5 Jan 16 04:01:42 hostname foreman[298891]: [298891] * Environment: production Jan 16 04:01:42 hostname foreman[298891]: [298891] * Master PID: 298891 Jan 16 04:01:42 hostname foreman[298891]: [298891] * Workers: 6 Jan 16 04:01:42 hostname foreman[298891]: [298891] * Restarts: (✔) hot (✖) phased Jan 16 04:01:42 hostname foreman[298891]: [298891] * Preloading application Jan 16 04:01:42 hostname foreman[298891]: [298891] * Activated unix:///run/foreman.sock Jan 16 04:01:42 hostname foreman[298891]: [298891] Use Ctrl-C to stop Jan 16 04:01:42 hostname foreman[298891]: [298891] * Starting control server on unix:///usr/share/foreman/tmp/sockets/pumactl.sock Jan 16 04:01:42 hostname foreman[298891]: [298891] Sending status to systemd every 1.0 sec Jan 16 04:01:43 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:01:43.053Z pid=298895 tid=6dqr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:19:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:01:43 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:01:43.053Z pid=298895 tid=6dqr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:20:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:01:43 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:01:43.053Z pid=298895 tid=6dqr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:36:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:01:43 hostname systemd[1]: Started Foreman jobs daemon - orchestrator on sidekiq. Jan 16 04:01:43 hostname foreman[298891]: [298891] - Worker 3 (PID: 299186) booted in 0.25s, phase: 0 Jan 16 04:01:43 hostname foreman[298891]: [298891] - Worker 2 (PID: 299180) booted in 0.36s, phase: 0 Jan 16 04:01:43 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:01:43.618Z pid=298895 tid=6dqr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:27:in `block in '", "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'"] Jan 16 04:01:43 hostname foreman[298891]: [298891] - Worker 0 (PID: 299176) booted in 0.53s, phase: 0 Jan 16 04:01:43 hostname foreman[298891]: [298891] - Worker 1 (PID: 299177) booted in 0.59s, phase: 0 Jan 16 04:01:43 hostname foreman[298891]: [298891] - Worker 5 (PID: 299193) booted in 0.85s, phase: 0 Jan 16 04:01:43 hostname foreman[298891]: [298891] - Worker 4 (PID: 299187) booted in 0.86s, phase: 0 Jan 16 04:01:44 hostname systemd[1]: Started Foreman. Jan 16 04:01:48 hostname server[298867]: Jan 16, 2024 4:01:48 AM org.apache.jasper.servlet.TldScanner scanJars Jan 16 04:01:48 hostname server[298867]: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Jan 16 04:01:49 hostname server[298867]: Jan 16, 2024 4:01:49 AM liquibase.database Jan 16 04:01:49 hostname server[298867]: INFO: Set default schema name to public Jan 16 04:01:52 hostname server[298867]: Jan 16, 2024 4:01:52 AM liquibase.executor Jan 16 04:01:52 hostname server[298867]: INFO: Changelog query completed. Jan 16 04:01:52 hostname server[298867]: Jan 16, 2024 4:01:52 AM liquibase.changelog Jan 16 04:01:52 hostname server[298867]: INFO: Reading from public.databasechangelog Jan 16 04:01:52 hostname server[298867]: Jan 16, 2024 4:01:52 AM liquibase.executor Jan 16 04:01:52 hostname server[298867]: INFO: Changelog query completed. Jan 16 04:01:52 hostname server[298867]: Jan 16, 2024 4:01:52 AM liquibase.executor Jan 16 04:01:52 hostname server[298867]: INFO: Changelog query completed. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.OwnerCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public void org.candlepin.model.CdnCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:53 hostname server[298867]: Jan 16, 2024 4:01:53 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:53 hostname server[298867]: WARNING: Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:54 hostname server[298867]: Jan 16, 2024 4:01:54 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:54 hostname server[298867]: WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:54 hostname server[298867]: Jan 16, 2024 4:01:54 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:54 hostname server[298867]: WARNING: Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:54 hostname server[298867]: Jan 16, 2024 4:01:54 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:54 hostname server[298867]: WARNING: Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:54 hostname server[298867]: Jan 16, 2024 4:01:54 AM com.google.inject.internal.ProxyFactory Jan 16 04:01:54 hostname server[298867]: WARNING: Method [public void org.candlepin.model.ContentCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@75ba3770]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Jan 16 04:01:58 hostname systemd[1]: Started puppetserver Service. Jan 16 04:01:58 hostname systemd[1]: Starting Foreman jobs daemon - worker-1 on sidekiq... Jan 16 04:01:58 hostname systemd[1]: Starting Foreman jobs daemon - worker-hosts-queue-1 on sidekiq... Jan 16 04:01:59 hostname dynflow-sidekiq@worker-hosts-queue-1[299469]: 2024-01-16T03:01:59.422Z pid=299469 tid=6frl INFO: Enabling systemd notification integration Jan 16 04:01:59 hostname dynflow-sidekiq@worker-1[299467]: 2024-01-16T03:01:59.516Z pid=299467 tid=6frr INFO: Enabling systemd notification integration Jan 16 04:02:00 hostname dynflow-sidekiq@worker-hosts-queue-1[299469]: 2024-01-16T03:02:00.226Z pid=299469 tid=6frl INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"} Jan 16 04:02:00 hostname dynflow-sidekiq@worker-hosts-queue-1[299469]: 2024-01-16T03:02:00.227Z pid=299469 tid=6frl INFO: GitLab reliable fetch activated! Jan 16 04:02:00 hostname dynflow-sidekiq@worker-1[299467]: 2024-01-16T03:02:00.529Z pid=299467 tid=6frr INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"} Jan 16 04:02:00 hostname dynflow-sidekiq@worker-1[299467]: 2024-01-16T03:02:00.541Z pid=299467 tid=6frr INFO: GitLab reliable fetch activated! Jan 16 04:02:04 hostname server[298867]: Jan 16, 2024 4:02:04 AM org.apache.catalina.startup.HostConfig deployDirectory Jan 16 04:02:04 hostname server[298867]: INFO: Deployment of web application directory [/var/lib/tomcat/webapps/candlepin] has finished in [33,865] ms Jan 16 04:02:04 hostname server[298867]: Jan 16, 2024 4:02:04 AM org.apache.coyote.AbstractProtocol start Jan 16 04:02:04 hostname server[298867]: INFO: Starting ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"] Jan 16 04:02:04 hostname server[298867]: Jan 16, 2024 4:02:04 AM org.apache.catalina.startup.Catalina start Jan 16 04:02:04 hostname server[298867]: INFO: Server startup in [34208] milliseconds Jan 16 04:02:08 hostname pulpcore-content[298868]: [16/Jan/2024:03:02:08 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:08 hostname pulpcore-content[298883]: [16/Jan/2024:03:02:08 +0000] "GET /pulp/content/cetin/REF/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 300 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:08 hostname pulpcore-content[298868]: [16/Jan/2024:03:02:08 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:09 hostname pulpcore-content[298861]: [16/Jan/2024:03:02:09 +0000] "GET /pulp/content/cetin/REF/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 300 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:09 hostname pulpcore-content[298909]: [16/Jan/2024:03:02:09 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:09 hostname pulpcore-content[298883]: [16/Jan/2024:03:02:09 +0000] "GET /pulp/content/cetin/REF/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:10 hostname pulpcore-content[298838]: [16/Jan/2024:03:02:09 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_3_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.3; generic; Linux.x86_64)" Jan 16 04:02:10 hostname pulpcore-content[298861]: [16/Jan/2024:03:02:10 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:10 hostname pulpcore-content[298838]: [16/Jan/2024:03:02:10 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:10 hostname pulpcore-content[298924]: [16/Jan/2024:03:02:10 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_3_x86_64/content/dist/rhel8/8.3/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.3; generic; Linux.x86_64)" Jan 16 04:02:10 hostname pulpcore-content[298833]: [16/Jan/2024:03:02:10 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_3_x86_64/content/dist/rhel8/8.3/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.3; generic; Linux.x86_64)" Jan 16 04:02:10 hostname pulpcore-content[298909]: [16/Jan/2024:03:02:10 +0000] "GET /pulp/content/cetin/TEST/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:11 hostname pulpcore-content[298834]: [16/Jan/2024:03:02:11 +0000] "GET /pulp/content/cetin/DEV/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:11 hostname pulpcore-content[298909]: [16/Jan/2024:03:02:11 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:12 hostname pulpcore-content[298909]: [16/Jan/2024:03:02:12 +0000] "GET /pulp/content/cetin/DEV/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:12 hostname pulpcore-content[298883]: [16/Jan/2024:03:02:12 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:12 hostname pulpcore-content[298868]: [16/Jan/2024:03:02:12 +0000] "GET /pulp/content/cetin/DEV/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:12 hostname pulpcore-content[298834]: [16/Jan/2024:03:02:12 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:20 hostname dynflow-sidekiq@worker-hosts-queue-1[299469]: 2024-01-16T03:02:20.733Z pid=299469 tid=6frl WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:19:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:02:20 hostname dynflow-sidekiq@worker-hosts-queue-1[299469]: 2024-01-16T03:02:20.733Z pid=299469 tid=6frl WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:22:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:02:20 hostname dynflow-sidekiq@worker-hosts-queue-1[299469]: 2024-01-16T03:02:20.733Z pid=299469 tid=6frl WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:36:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:02:20 hostname dynflow-sidekiq@worker-1[299467]: 2024-01-16T03:02:20.896Z pid=299467 tid=6frr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:19:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:02:20 hostname dynflow-sidekiq@worker-1[299467]: 2024-01-16T03:02:20.896Z pid=299467 tid=6frr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:22:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:02:20 hostname dynflow-sidekiq@worker-1[299467]: 2024-01-16T03:02:20.897Z pid=299467 tid=6frr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:36:in `'", "/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'"] Jan 16 04:02:20 hostname pulpcore-content[298861]: [16/Jan/2024:03:02:20 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:20 hostname dynflow-sidekiq@worker-hosts-queue-1[299469]: 2024-01-16T03:02:20.977Z pid=299469 tid=6frl WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:27:in `block in '", "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'"] Jan 16 04:02:20 hostname systemd[1]: Started Foreman jobs daemon - worker-hosts-queue-1 on sidekiq. Jan 16 04:02:21 hostname pulpcore-content[298883]: [16/Jan/2024:03:02:21 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:21 hostname dynflow-sidekiq@worker-1[299467]: 2024-01-16T03:02:21.164Z pid=299467 tid=6frr WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/foreman/extras/dynflow-sidekiq.rb:27:in `block in '", "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'"] Jan 16 04:02:21 hostname systemd[1]: Started Foreman jobs daemon - worker-1 on sidekiq. Jan 16 04:02:21 hostname dynflow-sidekiq@orchestrator[298895]: 2024-01-16T03:02:21.178Z pid=298895 tid=8cln class=Dynflow::Executors::Sidekiq::OrchestratorJobs::StartupComplete jid=8de66f6d67358ded24c14f69 WARN: `config.options[:key] = value` is deprecated, use `config[:key] = value`: ["/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow.rb:28:in `block in process_world'", "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:142:in `configure_server'"] Jan 16 04:02:21 hostname systemd[1]: Starting Foreman Proxy... Jan 16 04:02:21 hostname pulpcore-content[298924]: [16/Jan/2024:03:02:21 +0000] "GET /pulp/content/cetin/PROD/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:22 hostname systemd[1]: Started Foreman Proxy. Jan 16 04:02:36 hostname pulpcore-content[298904]: [16/Jan/2024:03:02:36 +0000] "GET /pulp/content/cetin/REF/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:36 hostname pulpcore-content[298868]: [16/Jan/2024:03:02:36 +0000] "GET /pulp/content/cetin/REF/Composite_RHEL8_8_x86_64/content/dist/rhel8/8.8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 299 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)" Jan 16 04:02:36 hostname pulpcore-content[298861]: [16/Jan/2024:03:02:36 +0000] "GET /pulp/content/cetin/REF/Composite_RHEL8_8_x86_64/custom/EPEL_-_Extra_Packages_for_Enterprise_Linux/EPEL8_x86_64/repodata/repomd.xml HTTP/1.1" 200 298 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)"