Sync status of products stops at 63%

Support
1

Problem:
we are currently trying to sync some products, including Amazon Linux SLES and RedHat repositories.
It is noticeable that with all syncs the task stops at 63% and does not run further.
In the production.log we see only one error which does not necessarily belong to this problem:

2023-07-24T17:16:32 [I|app|bb10ade8] Backtrace for ‘Action failed’ error (TypeError): incompatible marshal file format (can’t be read)

furthermore we see that for each sync 3 postgres pid’s are started, which completely load the server.

2810455 postgres 20 0 2592552 443576 420476 R 96,0 1,1 26:11.92 postmaster
2824606 postgres 20 0 2571732 377592 375024 R 96,0 0,9 29:36.19 postmaster
2815139 postgres 20 0 2571732 471812 469244 R 93,7 1,2 39:14.36 postmaster

Is there any other log I can take a look at, and maybe identify the problem?

Expected outcome:
successful syncing of products

Foreman and Proxy versions:
foreman-3.7.0-1.el8.noarch
katello-4.9.0-1.el8.noarch

Foreman and Proxy plugin versions:

Distribution and version:
RHEL8

1 Like
2

Check the dynflow console for the Foreman task to learn what Action the task is actually stuck on.

Most likely that action will be running some Pulp task, so you could try looking up that task in Pulp using something like:

pulp task show --href pulp_href

Where pulp_href is taken from your dynflow task output and should start with "/pulp/api/v3/tasks/…

This will tell you where the task started and ended, so you could search the system log for Pulp messages over this time range using something like:

journalctl --since "2023-07-24 08:20:18" --until "2023-07-24 10:44:44" -g pulp
1 Like
3

Thanks @quba42 !

I can see the following errors in the journalctl
Any hind about these errors?

Jul 24 16:34:20 katello.server pulpcore-content[2712644]:  [24/Jul/2023:14:34:20 +0000] "GET /pulp/content/CandA/amzlinux2_deve/redhat_linux_8/content/dist/rhel8/8/x86_64/codeready-builder/os/repodata/repomd.xml HTTP/1.1" 200 274 "-" "libdnf (Red Hat Enterprise Lin>
Jul 24 16:34:42 katello.server pulpcore-worker-7[2712492]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-7[2712492]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-7[2712492]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-7[2712492]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-7[2712492]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-7[2712492]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 294, in sleep
Jul 24 16:34:42 katello.server pulpcore-worker-3[2712486]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-3[2712486]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-3[2712486]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-3[2712486]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-3[2712486]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-3[2712486]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 294, in sleep
Jul 24 16:34:42 katello.server pulpcore-worker-6[2712491]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-6[2712491]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-6[2712491]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-6[2712491]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-6[2712491]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-6[2712491]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 294, in sleep
Jul 24 16:34:42 katello.server pulpcore-worker-4[2712484]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-4[2712484]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-4[2712484]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-4[2712484]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-4[2712484]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-4[2712484]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 294, in sleep
Jul 24 16:34:42 katello.server pulpcore-worker-2[2712487]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-2[2712487]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-2[2712487]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-2[2712487]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-2[2712487]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-2[2712487]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 294, in sleep
Jul 24 16:34:42 katello.server pulpcore-worker-1[2712473]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-1[2712473]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-1[2712473]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-1[2712473]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-1[2712473]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-1[2712473]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 294, in sleep
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 292, in sleep
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 165, in beat
Jul 24 16:34:42 katello.server pulpcore-worker-8[2712490]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 104, in handle_worker_heartbeat
Jul 24 16:34:42 katello.server pulpcore-worker-5[2712482]:   File "/usr/bin/pulpcore-worker", line 33, in <module>
Jul 24 16:34:42 katello.server pulpcore-worker-5[2712482]:     sys.exit(load_entry_point('pulpcore==3.22.7', 'console_scripts', 'pulpcore-worker')())
Jul 24 16:34:42 katello.server pulpcore-worker-5[2712482]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
Jul 24 16:34:42 katello.server pulpcore-worker-5[2712482]:     NewPulpWorker().run_forever()
Jul 24 16:34:42 katello.server pulpcore-worker-5[2712482]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in run_forever
Jul 24 16:34:42 katello.server pulpcore-worker-5[2712482]:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 294, in sleep
Jul 24 16:34:42 katello.server pulpcore-content[2712637]: pulp [None]: pulpcore.content:INFO: Content App '2712637@katello.server' failed to write a heartbeat to the database, sleeping for '7' seconds.
Jul 24 16:34:42 katello.server pulpcore-content[2712526]: pulp [None]: pulpcore.content:INFO: Content App '2712526@katello.server' failed to write a heartbeat to the database, sleeping for '7' seconds.
Jul 24 16:34:42 katello.server pulpcore-content[2712647]: pulp [None]: pulpcore.content:INFO: Content App '2712647@katello.server' failed to write a heartbeat to the database, sleeping for '7' seconds.
Jul 24 16:34:42 katello.server pulpcore-content[2712536]: pulp [None]: pulpcore.content:INFO: Content App '2712536@katello.server' failed to write a heartbeat to the database, sleeping for '7' seconds.
Jul 24 16:34:42 katello.server pulpcore-content[2712656]: pulp [None]: pulpcore.content:INFO: Content App '2712656@katello.server' failed to write a heartbeat to the database, sleeping for '7' seconds.
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@7.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@7.service: Failed with result 'exit-code'.
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@6.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@6.service: Failed with result 'exit-code'.
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@3.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@3.service: Failed with result 'exit-code'.
Jul 24 16:34:42 katello.server pulpcore-content[2712545]: pulp [None]: pulpcore.content:INFO: Content App '2712545@katello.server' failed to write a heartbeat to the database, sleeping for '7' seconds.
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@1.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 16:34:42 katello.server systemd[1]: pulpcore-worker@1.service: Failed with result 'exit-code'.
1 Like
4

I have not seen something like this before, but it looks a bit like the worker processes are failing to start.
Can you list the output of foreman-maintain service status?

1 Like
5

Sure here the output :slight_smile:

[root@katello ~]# foreman-maintain service status
Running Status Services
================================================================================
Get status of applicable services: 

Displaying the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
- displaying redis                                                              
● redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis.service.d
           └─90-limits.conf
   Active: active (running) since Mon 2023-07-24 14:36:21 CEST; 19h ago
  Process: 2711627 ExecStop=/usr/libexec/redis-shutdown (code=exited, status=0/SUCCESS)
 Main PID: 2712417 (redis-server)
   Status: "Ready to accept connections"
    Tasks: 5 (limit: 255042)
   Memory: 9.6M
   CGroup: /system.slice/redis.service
           └─2712417 /usr/bin/redis-server 127.0.0.1:6379

Jul 24 14:36:21 katello.server systemd[1]: Starting Redis persistent key-value database...
Jul 24 14:36:21 katello.server systemd[1]: Started Redis persistent key-value database.
\ displaying postgresql                                                         
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/postgresql.service.d
           └─postgresql.conf
   Active: active (running) since Mon 2023-07-24 16:35:44 CEST; 17h ago
  Process: 2798273 ExecStartPre=/usr/libexec/postgresql-check-db-dir postgresql (code=exited, status=0/SUCCESS)
 Main PID: 2798275 (postmaster)
    Tasks: 87 (limit: 255042)
   Memory: 6.8G
   CGroup: /system.slice/postgresql.service
           ├─2798275 /usr/bin/postmaster -D /var/lib/pgsql/data
           ├─2798277 postgres: logger   
           ├─2798281 postgres: checkpointer   
           ├─2798282 postgres: background writer   
           ├─2798283 postgres: walwriter   
           ├─2798284 postgres: autovacuum launcher   
           ├─2798285 postgres: stats collector   
           ├─2798286 postgres: logical replication launcher   
           ├─2798288 postgres: foreman foreman [local] idle
           ├─2798289 postgres: foreman foreman [local] idle
           ├─2798290 postgres: foreman foreman [local] idle
           ├─2798292 postgres: pulp pulpcore ::1(36442) idle
           ├─2798293 postgres: pulp pulpcore ::1(36452) idle
           ├─2798297 postgres: foreman foreman [local] idle
           ├─2798299 postgres: candlepin candlepin 127.0.0.1(53830) idle
           ├─2798300 postgres: candlepin candlepin 127.0.0.1(53846) idle
           ├─2798301 postgres: candlepin candlepin 127.0.0.1(53858) idle
           ├─2798302 postgres: pulp pulpcore ::1(36458) idle
           ├─2798310 postgres: foreman foreman [local] idle
           ├─2798312 postgres: foreman foreman [local] idle
           ├─2798321 postgres: pulp pulpcore ::1(36510) idle
           ├─2798323 postgres: pulp pulpcore ::1(36526) idle
           ├─2798327 postgres: pulp pulpcore ::1(36536) idle
           ├─2798328 postgres: pulp pulpcore ::1(36540) idle
           ├─2798329 postgres: pulp pulpcore ::1(36556) idle
           ├─2798331 postgres: pulp pulpcore ::1(36570) idle
           ├─2798332 postgres: pulp pulpcore ::1(36578) idle
           ├─2798333 postgres: foreman foreman [local] idle
           ├─2798336 postgres: pulp pulpcore ::1(36592) idle
           ├─2798338 postgres: pulp pulpcore ::1(36604) idle
           ├─2798341 postgres: pulp pulpcore ::1(36620) idle
           ├─2798342 postgres: pulp pulpcore ::1(36634) idle
           ├─2798344 postgres: foreman foreman [local] SELECT
           ├─2798353 postgres: pulp pulpcore ::1(36640) idle
           ├─2798354 postgres: pulp pulpcore ::1(36656) idle
           ├─2798362 postgres: pulp pulpcore ::1(36664) idle
           ├─2798366 postgres: pulp pulpcore ::1(36678) idle
           ├─2798367 postgres: pulp pulpcore ::1(36690) idle
           ├─2798368 postgres: pulp pulpcore ::1(36694) idle
           ├─2798407 postgres: pulp pulpcore ::1(47664) idle
           ├─2798410 postgres: foreman foreman [local] idle
           ├─2798460 postgres: pulp pulpcore ::1(47670) idle
           ├─2798461 postgres: pulp pulpcore ::1(47678) idle
           ├─2798462 postgres: pulp pulpcore ::1(47682) idle
           ├─2798465 postgres: pulp pulpcore ::1(47692) idle
           ├─2798468 postgres: foreman foreman [local] idle
           ├─2798473 postgres: foreman foreman [local] idle
           ├─2798483 postgres: foreman foreman [local] idle
           ├─2798484 postgres: foreman foreman [local] idle
           ├─2798485 postgres: foreman foreman [local] idle
           ├─2798486 postgres: foreman foreman [local] idle
           ├─2798487 postgres: foreman foreman [local] idle
           ├─2798488 postgres: foreman foreman [local] idle
           ├─2798490 postgres: foreman foreman [local] idle
           ├─2798491 postgres: foreman foreman [local] idle
           ├─2798492 postgres: foreman foreman [local] idle
           ├─2798493 postgres: foreman foreman [local] idle
           ├─2798497 postgres: foreman foreman [local] idle
           ├─2798502 postgres: foreman foreman [local] idle
           ├─2802956 postgres: foreman foreman [local] idle
           ├─2810490 postgres: foreman foreman [local] SELECT
           ├─2815138 postgres: parallel worker for PID 2810490   
           ├─2815139 postgres: parallel worker for PID 2810490   
           ├─2819847 postgres: foreman foreman [local] idle
           ├─2824605 postgres: parallel worker for PID 2798344   
           ├─2824606 postgres: parallel worker for PID 2798344   
           ├─2824907 postgres: foreman foreman [local] idle
           ├─2847025 postgres: foreman foreman [local] idle
           ├─2847063 postgres: foreman foreman [local] idle
           ├─3317888 postgres: foreman foreman [local] idle
           ├─3381054 postgres: foreman foreman [local] SELECT
           ├─3385027 postgres: foreman foreman [local] idle
           ├─3385030 postgres: foreman foreman [local] SELECT
           ├─3385031 postgres: foreman foreman [local] SELECT
           ├─3386105 postgres: parallel worker for PID 3385030   
           ├─3386106 postgres: parallel worker for PID 3385030   
           ├─3386422 postgres: parallel worker for PID 3381054   
           ├─3409836 postgres: foreman foreman [local] idle
           ├─3440351 postgres: foreman foreman [local] idle
           ├─3448009 postgres: foreman foreman [local] idle
           ├─3534992 postgres: foreman foreman [local] idle
           ├─3538908 postgres: candlepin candlepin 127.0.0.1(39282) idle
           ├─3541152 postgres: foreman foreman [local] idle
           ├─3542431 postgres: candlepin candlepin 127.0.0.1(47634) idle
           ├─3542432 postgres: candlepin candlepin 127.0.0.1(47648) idle
           ├─3544832 postgres: candlepin candlepin 127.0.0.1(47314) idle
           └─3544833 postgres: candlepin candlepin 127.0.0.1(47318) idle

Jul 24 16:35:44 katello.server systemd[1]: Starting PostgreSQL database server...
Jul 24 16:35:44 katello.server postmaster[2798275]: 2023-07-24 16:35:44 CEST LOG:  PostgreSQL 12.12 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10), 64-bit startet
Jul 24 16:35:44 katello.server postmaster[2798275]: 2023-07-24 16:35:44 CEST LOG:  erwarte Verbindungen auf IPv6-Adresse »::1«, Port 5432
Jul 24 16:35:44 katello.server postmaster[2798275]: 2023-07-24 16:35:44 CEST LOG:  erwarte Verbindungen auf IPv4-Adresse »127.0.0.1«, Port 5432
Jul 24 16:35:44 katello.server postmaster[2798275]: 2023-07-24 16:35:44 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/var/run/postgresql/.s.PGSQL.5432«
Jul 24 16:35:44 katello.server postmaster[2798275]: 2023-07-24 16:35:44 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/tmp/.s.PGSQL.5432«
Jul 24 16:35:44 katello.server postmaster[2798275]: 2023-07-24 16:35:44 CEST LOG:  Logausgabe wird an Logsammelprozess umgeleitet
Jul 24 16:35:44 katello.server postmaster[2798275]: 2023-07-24 16:35:44 CEST TIPP:  Die weitere Logausgabe wird im Verzeichnis »log« erscheinen.
Jul 24 16:35:44 katello.server systemd[1]: Started PostgreSQL database server.
\ displaying pulpcore-api                                                       
● pulpcore-api.service - Pulp API Server
   Loaded: loaded (/etc/systemd/system/pulpcore-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 14:36:27 CEST; 19h ago
 Main PID: 2712436 (gunicorn)
   Status: "Gunicorn arbiter booted"
    Tasks: 6 (limit: 255042)
   Memory: 650.4M
   CGroup: /system.slice/pulpcore-api.service
           ├─2712436 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 50 --max-requests-jitter 30 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─2825212 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 50 --max-requests-jitter 30 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─3385687 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 50 --max-requests-jitter 30 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─3385696 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 50 --max-requests-jitter 30 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           ├─3386094 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 50 --max-requests-jitter 30 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
           └─3386107 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.app.wsgi:application --preload --timeout 90 --workers 5 --max-requests 50 --max-requests-jitter 30 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"

Jul 25 06:06:45 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:06:45 +0000] "GET /pulp/api/v3/content/rpm/packageenvironments/?limit=2000&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F2bae7c78-fc2d-4fc6-b9e6-37fdbc8727e4%2Fversions%2F520%2F HTTP/1.1" 200 15396 "-" "OpenAPI-Generator/3.19.6/ruby"
Jul 25 06:06:45 katello.server pulpcore-api[3386094]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:06:45 +0000] "GET /pulp/api/v3/content/rpm/modulemd_defaults/?limit=2000&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F2bae7c78-fc2d-4fc6-b9e6-37fdbc8727e4%2Fversions%2F520%2F HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.19.6/ruby"
Jul 25 06:06:46 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:06:46 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/dc77dfb7-cb92-4649-8654-20656706359a/modify/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.19.6/ruby"
Jul 25 06:06:57 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:06:57 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/dc77dfb7-cb92-4649-8654-20656706359a/modify/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.19.6/ruby"
Jul 25 06:07:04 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:07:04 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/dc77dfb7-cb92-4649-8654-20656706359a/modify/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.19.6/ruby"
Jul 25 06:07:04 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:07:04 +0000] "GET /pulp/api/v3/tasks/13f88d52-c771-4a2a-b07c-8098bd59e1ff/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.22.4/ruby"
Jul 25 06:07:04 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:07:04 +0000] "GET /pulp/api/v3/tasks/52327f7e-2974-47a5-98ae-994996ba27ba/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.22.4/ruby"
Jul 25 06:07:05 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:07:05 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/1d2d326d-227e-47fd-85e2-4b2d076e2dad/ HTTP/1.1" 200 521 "-" "OpenAPI-Generator/3.19.6/ruby"
Jul 25 06:07:05 katello.server pulpcore-api[3385687]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]:  - - [25/Jul/2023:04:07:05 +0000] "GET /pulp/api/v3/content/rpm/packages/?arch__ne=src&fields=pulp_href&limit=2000&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2Feb17a265-a2f4-4f46-a844-5dd4e4c50f37%2Fversions%2F34%2F HTTP/1.1" 200 1372 "-" "OpenAPI-Generator/3.19.6/ruby"
Jul 25 07:34:06 katello.server pulpcore-api[3386094]: pulp [6397fdfb07f84396ad73930eac358a94]:  - - [25/Jul/2023:05:34:06 +0000] "GET /pulp/api/v3/docs/api.json HTTP/1.1" 200 2588030 "-" "Pulp-CLI/0.14.0"
\ displaying pulpcore-content                                                   
● pulpcore-content.service - Pulp Content App
   Loaded: loaded (/etc/systemd/system/pulpcore-content.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 14:36:27 CEST; 19h ago
 Main PID: 2712435 (gunicorn)
   Status: "Gunicorn arbiter booted"
    Tasks: 52 (limit: 255042)
   Memory: 2.1G
   CGroup: /system.slice/pulpcore-content.service
           ├─2712435 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712455 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712456 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712471 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712495 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712505 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712511 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712517 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712526 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712536 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712545 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712637 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712640 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712644 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712647 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712649 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           ├─2712655 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -
           └─2712656 /usr/bin/python3.9 /usr/bin/gunicorn pulpcore.content:server --preload --timeout 90 --worker-class aiohttp.GunicornWebWorker --workers 17 --access-logfile -

Jul 25 09:46:10 katello.server pulpcore-content[2712655]:  [25/Jul/2023:07:46:10 +0000] "GET /pulp/content/CandA/amzlinux2_prod/Amazon_Linux_2/custom/amazon-linux2/amazon-linux2-python38/repodata/repomd.xml HTTP/1.1" 200 273 "-" "urlgrabber/3.10 yum/3.4.3"
Jul 25 09:46:11 katello.server pulpcore-content[2712495]:  [25/Jul/2023:07:46:10 +0000] "GET /pulp/content/CandA/amzlinux2_prod/Amazon_Linux_2/custom/amazon-subscriptionmanager/amazon-subscriptionmanager/repodata/repomd.xml HTTP/1.1" 200 273 "-" "urlgrabber/3.10 yum/3.4.3"
Jul 25 09:46:11 katello.server pulpcore-content[2712655]:  [25/Jul/2023:07:46:11 +0000] "GET /pulp/content/CandA/amzlinux2_prod/Amazon_Linux_2/custom/hashicorp/hashicorp/repodata/repomd.xml HTTP/1.1" 200 273 "-" "urlgrabber/3.10 yum/3.4.3"
Jul 25 09:46:11 katello.server pulpcore-content[2712655]:  [25/Jul/2023:07:46:11 +0000] "GET /pulp/content/CandA/amzlinux2_prod/Amazon_Linux_2/custom/jenkins_redhat_stable/Jenkins/repodata/repomd.xml HTTP/1.1" 200 273 "-" "urlgrabber/3.10 yum/3.4.3"
Jul 25 09:46:11 katello.server pulpcore-content[2712656]:  [25/Jul/2023:07:46:11 +0000] "GET /pulp/content/CandA/amzlinux2_prod/Amazon_Linux_2/custom/katello-client_new/katello-client_new/repodata/repomd.xml HTTP/1.1" 200 273 "-" "urlgrabber/3.10 yum/3.4.3"
Jul 25 09:46:11 katello.server pulpcore-content[2712655]:  [25/Jul/2023:07:46:11 +0000] "GET /pulp/content/CandA/amzlinux2_prod/Amazon_Linux_2/custom/symas-openldap/symas-openldap-rhel7/repodata/repomd.xml HTTP/1.1" 200 273 "-" "urlgrabber/3.10 yum/3.4.3"
Jul 25 09:46:24 katello.server pulpcore-content[2712655]:  [25/Jul/2023:07:46:24 +0000] "GET /pulp/content/CandA/amzlinux2_deve/redhat_linux_8/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 274 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)"
Jul 25 09:46:24 katello.server pulpcore-content[2712656]:  [25/Jul/2023:07:46:24 +0000] "GET /pulp/content/CandA/amzlinux2_deve/redhat_linux_8/content/dist/layered/rhel8/x86_64/sat-tools/6.8/os/repodata/repomd.xml HTTP/1.1" 200 273 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)"
Jul 25 09:46:24 katello.server pulpcore-content[2712455]:  [25/Jul/2023:07:46:24 +0000] "GET /pulp/content/CandA/amzlinux2_deve/redhat_linux_8/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 274 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)"
Jul 25 09:46:25 katello.server pulpcore-content[2712656]:  [25/Jul/2023:07:46:25 +0000] "GET /pulp/content/CandA/amzlinux2_deve/redhat_linux_8/content/dist/rhel8/8/x86_64/codeready-builder/os/repodata/repomd.xml HTTP/1.1" 200 274 "-" "libdnf (Red Hat Enterprise Linux 8.8; generic; Linux.x86_64)"
/ displaying pulpcore-worker@1.service                                          
● pulpcore-worker@1.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:42 CEST; 17h ago
 Main PID: 2798249 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 97.8M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
           └─2798249 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:31 katello.server pulpcore-worker-1[3385616]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task ccc48e1f-2de1-4a8a-944f-b55fb10a5bff
Jul 25 06:02:31 katello.server pulpcore-worker-1[3385616]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Amazon_Linux_2-NodeJS_new, version=39
Jul 25 06:02:54 katello.server pulpcore-worker-1[3385616]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: cac39f72-c416-4989-8649-2610923ed27a created
Jul 25 06:02:54 katello.server pulpcore-worker-1[3385616]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed ccc48e1f-2de1-4a8a-944f-b55fb10a5bff
Jul 25 06:03:13 katello.server pulpcore-worker-1[3385903]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task e7284962-75ec-4dcf-b87a-b9f341f39dfe
Jul 25 06:03:14 katello.server pulpcore-worker-1[3385903]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed e7284962-75ec-4dcf-b87a-b9f341f39dfe
Jul 25 06:06:46 katello.server pulpcore-worker-1[3389602]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 819a999d-b816-484b-b0a0-5120d809c159
Jul 25 06:07:07 katello.server pulpcore-worker-1[3389602]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 819a999d-b816-484b-b0a0-5120d809c159
Jul 25 06:07:08 katello.server pulpcore-worker-1[3389728]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 13f88d52-c771-4a2a-b07c-8098bd59e1ff
Jul 25 06:07:23 katello.server pulpcore-worker-1[3389728]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 13f88d52-c771-4a2a-b07c-8098bd59e1ff
/ displaying pulpcore-worker@2.service                                          
● pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:42 CEST; 17h ago
 Main PID: 2798252 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 100.1M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
           └─2798252 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:30 katello.server pulpcore-worker-2[3385595]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 18645ecd-adcf-4315-848b-ab000d03f08b
Jul 25 06:02:30 katello.server pulpcore-worker-2[3385595]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Amazon_Linux_2-Jenkins, version=44
Jul 25 06:02:49 katello.server pulpcore-worker-2[3385595]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: 471371d6-92e7-432d-b53f-00fd06ed8659 created
Jul 25 06:02:50 katello.server pulpcore-worker-2[3385595]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 18645ecd-adcf-4315-848b-ab000d03f08b
Jul 25 06:02:50 katello.server pulpcore-worker-2[3385715]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 49ab92f6-da57-4322-ab11-ecde861eead1
Jul 25 06:02:50 katello.server pulpcore-worker-2[3385715]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=NodeJS_14-24446339, version=36
Jul 25 06:03:06 katello.server pulpcore-worker-2[3385715]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: 1e0c83e2-46b3-47f3-a8dc-8a8f67e704e1 created
Jul 25 06:03:06 katello.server pulpcore-worker-2[3385715]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 49ab92f6-da57-4322-ab11-ecde861eead1
Jul 25 06:03:21 katello.server pulpcore-worker-2[3386046]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task a2241c20-f459-4229-8729-53f0adda923b
Jul 25 06:03:22 katello.server pulpcore-worker-2[3386046]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed a2241c20-f459-4229-8729-53f0adda923b
/ displaying pulpcore-worker@3.service                                          
● pulpcore-worker@3.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:42 CEST; 17h ago
 Main PID: 2798250 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 101.9M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@3.service
           └─2798250 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:12 katello.server pulpcore-worker-3[3385380]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 49f72702-ad50-45fe-9c93-21f554594d0d
Jul 25 06:02:13 katello.server pulpcore-worker-3[3385380]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 49f72702-ad50-45fe-9c93-21f554594d0d
Jul 25 06:02:22 katello.server pulpcore-worker-3[3385536]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 7cb3ffb7-e27f-406b-9208-a80ad5e2bd94
Jul 25 06:02:22 katello.server pulpcore-worker-3[3385536]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 7cb3ffb7-e27f-406b-9208-a80ad5e2bd94
Jul 25 06:02:29 katello.server pulpcore-worker-3[3385581]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task fe5fd20e-0419-4ae2-b7b9-76bfd9310d0a
Jul 25 06:02:29 katello.server pulpcore-worker-3[3385581]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Amazon_Linux_2-hashicorp, version=47
Jul 25 06:02:53 katello.server pulpcore-worker-3[3385581]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: ddc3b4d8-e21f-4936-9ec4-2752ac782748 created
Jul 25 06:02:53 katello.server pulpcore-worker-3[3385581]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed fe5fd20e-0419-4ae2-b7b9-76bfd9310d0a
Jul 25 06:03:22 katello.server pulpcore-worker-3[3386059]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 75f1bd9f-f9e7-4d04-8956-73a66918b186
Jul 25 06:03:23 katello.server pulpcore-worker-3[3386059]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 75f1bd9f-f9e7-4d04-8956-73a66918b186
/ displaying pulpcore-worker@4.service                                          
● pulpcore-worker@4.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:42 CEST; 17h ago
 Main PID: 2798253 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 100.7M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
           └─2798253 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:30 katello.server pulpcore-worker-4[3385602]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 967a5fdf-e1d0-4785-9366-012ab79c8706
Jul 25 06:02:30 katello.server pulpcore-worker-4[3385602]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Amazon_Linux_2-amazon-linux2-docker-new, version=39
Jul 25 06:02:50 katello.server pulpcore-worker-4[3385602]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: b6118ef2-c82b-4055-97aa-111195d678c2 created
Jul 25 06:02:50 katello.server pulpcore-worker-4[3385602]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 967a5fdf-e1d0-4785-9366-012ab79c8706
Jul 25 06:02:50 katello.server pulpcore-worker-4[3385719]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task fc9419e9-2f34-4343-b375-75571a9761a3
Jul 25 06:02:50 katello.server pulpcore-worker-4[3385719]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=amazon-linux2-postgresql10-24561798, version=37
Jul 25 06:03:02 katello.server pulpcore-worker-4[3385719]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: ebf76eb2-fbb8-4840-be88-99ef57572121 created
Jul 25 06:03:02 katello.server pulpcore-worker-4[3385719]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed fc9419e9-2f34-4343-b375-75571a9761a3
Jul 25 06:03:20 katello.server pulpcore-worker-4[3385988]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 1f4fef64-2eb9-4a47-8eee-4391d61c8c01
Jul 25 06:03:21 katello.server pulpcore-worker-4[3385988]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 1f4fef64-2eb9-4a47-8eee-4391d61c8c01
/ displaying pulpcore-worker@5.service                                          
● pulpcore-worker@5.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:42 CEST; 17h ago
 Main PID: 2798251 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 100.9M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@5.service
           └─2798251 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:28 katello.server pulpcore-worker-5[3385574]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task d38ae32d-df39-4fd5-9bca-d6aa6bd09983
Jul 25 06:02:28 katello.server pulpcore-worker-5[3385574]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Amazon_Linux_2-amazon-subscriptionmanager, version=42
Jul 25 06:02:55 katello.server pulpcore-worker-5[3385574]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: beb8e02f-b71b-4b88-b413-92bb04ff9ac2 created
Jul 25 06:02:55 katello.server pulpcore-worker-5[3385574]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed d38ae32d-df39-4fd5-9bca-d6aa6bd09983
Jul 25 06:03:15 katello.server pulpcore-worker-5[3385934]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 5a9ebffc-6139-4e9d-b241-3c2d36db8f07
Jul 25 06:03:16 katello.server pulpcore-worker-5[3385934]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 5a9ebffc-6139-4e9d-b241-3c2d36db8f07
Jul 25 06:03:17 katello.server pulpcore-worker-5[3385947]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 5d22ca26-2e4b-41f0-aa10-c74537fdb920
Jul 25 06:03:17 katello.server pulpcore-worker-5[3385947]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 5d22ca26-2e4b-41f0-aa10-c74537fdb920
Jul 25 06:04:44 katello.server pulpcore-worker-5[3386370]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task d2fbfd6e-2775-451e-8d20-cd97396683cd
Jul 25 06:05:27 katello.server pulpcore-worker-5[3386370]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed d2fbfd6e-2775-451e-8d20-cd97396683cd
/ displaying pulpcore-worker@6.service                                          
● pulpcore-worker@6.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:42 CEST; 17h ago
 Main PID: 2798254 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 101.8M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@6.service
           └─2798254 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:19 katello.server pulpcore-worker-6[3385465]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task ab9d91f3-545a-4a09-a9f4-fa1184ec6855
Jul 25 06:02:20 katello.server pulpcore-worker-6[3385465]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed ab9d91f3-545a-4a09-a9f4-fa1184ec6855
Jul 25 06:02:31 katello.server pulpcore-worker-6[3385609]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 5409bf46-8093-414c-9842-618b2e2b6677
Jul 25 06:02:31 katello.server pulpcore-worker-6[3385609]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Amazon_Linux_2-amazon-linux2-openjdk11-new, version=39
Jul 25 06:02:58 katello.server pulpcore-worker-6[3385609]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: 38325c90-b0e5-44d4-b8f6-0832f7d4d3aa created
Jul 25 06:02:58 katello.server pulpcore-worker-6[3385609]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 5409bf46-8093-414c-9842-618b2e2b6677
Jul 25 06:03:18 katello.server pulpcore-worker-6[3385960]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 4d276077-6c7a-4d14-b9a0-2f6dd3fa82f3
Jul 25 06:03:18 katello.server pulpcore-worker-6[3385960]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 4d276077-6c7a-4d14-b9a0-2f6dd3fa82f3
Jul 25 09:28:27 katello.server pulpcore-worker-6[2798254]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean missing worker 3764882@katello.server.
Jul 25 09:28:27 katello.server pulpcore-worker-6[2798254]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean missing worker 3764923@katello.server.
/ displaying pulpcore-worker@7.service                                          
● pulpcore-worker@7.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:41 CEST; 17h ago
 Main PID: 2798248 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 100.9M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@7.service
           └─2798248 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:50 katello.server pulpcore-worker-7[3385728]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 6c2e2db4-c8ce-45a1-8ff0-33f41c33c964
Jul 25 06:02:51 katello.server pulpcore-worker-7[3385728]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=katello-client_new-25013384, version=34
Jul 25 06:02:59 katello.server pulpcore-worker-7[3385728]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: ae100f00-b7a6-4031-8d4c-ecf7c0ed4553 created
Jul 25 06:02:59 katello.server pulpcore-worker-7[3385728]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 6c2e2db4-c8ce-45a1-8ff0-33f41c33c964
Jul 25 06:03:14 katello.server pulpcore-worker-7[3385916]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task d094e0eb-77c8-4480-be10-7b4874b20213
Jul 25 06:03:15 katello.server pulpcore-worker-7[3385916]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed d094e0eb-77c8-4480-be10-7b4874b20213
Jul 25 06:03:23 katello.server pulpcore-worker-7[3386074]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task fe64a2b4-edca-4c08-afc2-0aad8f2bc97f
Jul 25 06:03:24 katello.server pulpcore-worker-7[3386074]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed fe64a2b4-edca-4c08-afc2-0aad8f2bc97f
Jul 25 06:07:23 katello.server pulpcore-worker-7[3389775]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 52327f7e-2974-47a5-98ae-994996ba27ba
Jul 25 06:08:28 katello.server pulpcore-worker-7[3389775]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 52327f7e-2974-47a5-98ae-994996ba27ba
/ displaying pulpcore-worker@8.service                                          
● pulpcore-worker@8.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:35:46 CEST; 17h ago
 Main PID: 2798343 (pulpcore-worker)
    Tasks: 1 (limit: 255042)
   Memory: 110.7M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@8.service
           └─2798343 /usr/bin/python3.9 /usr/bin/pulpcore-worker

Jul 25 06:02:51 katello.server pulpcore-worker-8[3385736]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task f6a9ecd8-3a2a-4d67-a5af-623644befa90
Jul 25 06:02:51 katello.server pulpcore-worker-8[3385736]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=amazon-linux2-python38-45835129, version=17
Jul 25 06:03:04 katello.server pulpcore-worker-8[3385736]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulp_rpm.app.tasks.publishing:INFO: Publication: e213a4c1-5566-4610-8fae-adb60474dd48 created
Jul 25 06:03:04 katello.server pulpcore-worker-8[3385736]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed f6a9ecd8-3a2a-4d67-a5af-623644befa90
Jul 25 06:03:11 katello.server pulpcore-worker-8[3385874]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 8694cd06-69f6-4a05-8bee-bdd4c96d87a5
Jul 25 06:03:11 katello.server pulpcore-worker-8[3385874]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 8694cd06-69f6-4a05-8bee-bdd4c96d87a5
Jul 25 06:03:12 katello.server pulpcore-worker-8[3385888]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task f56ce7ce-a997-4e33-bf56-cc2f37fa1dc4
Jul 25 06:03:12 katello.server pulpcore-worker-8[3385888]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed f56ce7ce-a997-4e33-bf56-cc2f37fa1dc4
Jul 25 06:03:19 katello.server pulpcore-worker-8[3385973]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Starting task ba434fe5-fbea-4c61-b2ef-408d2ced58b8
Jul 25 06:03:19 katello.server pulpcore-worker-8[3385973]: pulp [6faa7cdd-3792-44da-b635-933e1739d2c4]: pulpcore.tasking.pulpcore_worker:INFO: Task completed ba434fe5-fbea-4c61-b2ef-408d2ced58b8
/ displaying tomcat                                                             
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 14:36:27 CEST; 19h ago
 Main PID: 2712493 (java)
    Tasks: 98 (limit: 255042)
   Memory: 1.4G
   CGroup: /system.slice/tomcat.service
           └─2712493 /usr/lib/jvm/jre-17/bin/java -Xms1024m -Xmx4096m -Dcom.redhat.fips=false -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Jul 24 14:36:48 katello.server server[2712493]: 24-Jul-2023 14:36:48.426 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2f274bd7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 24 14:36:48 katello.server server[2712493]: 24-Jul-2023 14:36:48.461 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.CdnCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2f274bd7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 24 14:36:48 katello.server server[2712493]: 24-Jul-2023 14:36:48.471 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2f274bd7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 24 14:36:48 katello.server server[2712493]: 24-Jul-2023 14:36:48.514 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2f274bd7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 24 14:36:48 katello.server server[2712493]: 24-Jul-2023 14:36:48.514 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2f274bd7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 24 14:36:48 katello.server server[2712493]: 24-Jul-2023 14:36:48.561 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2f274bd7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 24 14:36:48 katello.server server[2712493]: 24-Jul-2023 14:36:48.584 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ContentCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2f274bd7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 24 14:36:59 katello.server server[2712493]: 24-Jul-2023 14:36:59.847 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/candlepin] has finished in [24,279] ms
Jul 24 14:36:59 katello.server server[2712493]: 24-Jul-2023 14:36:59.850 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
Jul 24 14:36:59 katello.server server[2712493]: 24-Jul-2023 14:36:59.881 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [24407] milliseconds
- displaying dynflow-sidekiq@orchestrator                                       
● dynflow-sidekiq@orchestrator.service - Foreman jobs daemon - orchestrator on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 14:36:57 CEST; 19h ago
     Docs: https://theforeman.org
 Main PID: 2712504 (sidekiq)
   Status: "Everything ready for world: 439975e2-b095-4246-aa00-dad770a9af62"
    Tasks: 10 (limit: 255042)
   Memory: 484.4M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@orchestrator.service
           └─2712504 sidekiq 6.3.1  [0 of 1 busy]

Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/execution_plan.rb:217:in `prepare_for_rescue'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/director.rb:290:in `rescue!'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/director.rb:260:in `try_to_rescue'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/director.rb:253:in `unless_done'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/director.rb:215:in `work_finished'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/executors/abstract/core.rb:54:in `work_finished'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/executors/sidekiq/core.rb:71:in `work_finished'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: [ concurrent-ruby ]
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/executors/abstract/core.rb:122:in `on_message'
Jul 24 16:35:49 katello.server dynflow-sidekiq@orchestrator[2712504]: [ concurrent-ruby ]
- displaying foreman                                                            
● foreman.service - Foreman
   Loaded: loaded (/usr/lib/systemd/system/foreman.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/foreman.service.d
           └─installer.conf
   Active: active (running) since Mon 2023-07-24 14:36:58 CEST; 19h ago
     Docs: https://theforeman.org
 Main PID: 2712509 (rails)
   Status: "Puma 6.2.2: cluster: 12/12, worker_status: [{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 4 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog }]"
    Tasks: 217 (limit: 255042)
   Memory: 6.4G
   CGroup: /system.slice/foreman.service
           ├─2712509 puma 6.2.2 (unix:///run/foreman.sock) [foreman]
           ├─2714652 puma: cluster worker 0: 2712509 [foreman]
           ├─2714657 puma: cluster worker 1: 2712509 [foreman]
           ├─2714675 puma: cluster worker 2: 2712509 [foreman]
           ├─2714686 puma: cluster worker 3: 2712509 [foreman]
           ├─2714701 puma: cluster worker 4: 2712509 [foreman]
           ├─2714705 puma: cluster worker 5: 2712509 [foreman]
           ├─2714716 puma: cluster worker 6: 2712509 [foreman]
           ├─2714727 puma: cluster worker 7: 2712509 [foreman]
           ├─2714740 puma: cluster worker 8: 2712509 [foreman]
           ├─2714754 puma: cluster worker 9: 2712509 [foreman]
           ├─2714769 puma: cluster worker 10: 2712509 [foreman]
           └─2714782 puma: cluster worker 11: 2712509 [foreman]

Jul 24 16:43:40 katello.server foreman[2714701]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/dispatcher/client_dispatcher.rb:116:in `publish_request'
Jul 24 16:43:40 katello.server foreman[2714701]: [ concurrent-ruby ]
Jul 24 16:43:40 katello.server foreman[2714701]: E, [2023-07-24T16:43:40.359561 #2714701] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: 0e7f6d27-ac99-479d-9c65-5a5b11a43460-7, sender_id: 0e7f6d27-ac99-479d-9c65-5a5b11a43460, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 1d4863a7-25d8-42ba-bfa4-a7fb71b35b65, step_id: 3, event: #<Actions::ProxyAction::CallbackData:0x000055d36d119860>, time: , optional: false]] (Dynflow::Error)
Jul 24 16:43:40 katello.server foreman[2714701]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/dispatcher/client_dispatcher.rb:149:in `dispatch_request'
Jul 24 16:43:40 katello.server foreman[2714701]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/dispatcher/client_dispatcher.rb:118:in `block (2 levels) in publish_request'
Jul 24 16:43:40 katello.server foreman[2714701]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/dispatcher/client_dispatcher.rb:212:in `track_request'
Jul 24 16:43:40 katello.server foreman[2714701]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/dispatcher/client_dispatcher.rb:117:in `block in publish_request'
Jul 24 16:43:40 katello.server foreman[2714701]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/dispatcher/client_dispatcher.rb:254:in `with_ping_request_caching'
Jul 24 16:43:40 katello.server foreman[2714701]: /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/dispatcher/client_dispatcher.rb:116:in `publish_request'
Jul 24 16:43:40 katello.server foreman[2714701]: [ concurrent-ruby ]
- displaying httpd                                                              
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 16:38:03 CEST; 17h ago
     Docs: man:httpd.service(8)
  Process: 961611 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 2802765 (httpd)
   Status: "Total requests: 11456; Idle/Busy workers 100/0;Requests/sec: 0.186; Bytes served/sec:  20KB/sec"
    Tasks: 85 (limit: 255042)
   Memory: 61.0M
   CGroup: /system.slice/httpd.service
           ├─2802765 /usr/sbin/httpd -DFOREGROUND
           ├─2802766 /usr/sbin/httpd -DFOREGROUND
           └─3455646 /usr/sbin/httpd -DFOREGROUND

Jul 24 16:38:03 katello.server systemd[1]: Starting The Apache HTTP Server...
Jul 24 16:38:03 katello.server systemd[1]: Started The Apache HTTP Server.
Jul 24 16:38:03 katello.server httpd[2802765]: Server configured, listening on: port 80, port 443
| displaying dynflow-sidekiq@worker-1                                           
● dynflow-sidekiq@worker-1.service - Foreman jobs daemon - worker-1 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 14:37:19 CEST; 19h ago
     Docs: https://theforeman.org
 Main PID: 2715034 (sidekiq)
   Status: "Everything ready for world: 822dd51d-7ef4-4cc3-a9df-6fe327f4d045"
    Tasks: 14 (limit: 255042)
   Memory: 1.0G
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-1.service
           └─2715034 sidekiq 6.3.1  [5 of 5 busy]

Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
Jul 24 16:35:51 katello.server dynflow-sidekiq@worker-1[2715034]: /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
| displaying dynflow-sidekiq@worker-hosts-queue-1                               
● dynflow-sidekiq@worker-hosts-queue-1.service - Foreman jobs daemon - worker-hosts-queue-1 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-24 14:37:19 CEST; 19h ago
     Docs: https://theforeman.org
 Main PID: 2715035 (sidekiq)
   Status: "Everything ready for world: 7fc5bf88-cfca-4735-8250-ebab0dffb34a"
    Tasks: 14 (limit: 255042)
   Memory: 438.1M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-hosts-queue-1.service
           └─2715035 sidekiq 6.3.1  [0 of 5 busy]

Jul 24 16:34:42 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         from /usr/share/gems/gems/sequel-5.68.0/lib/sequel/adapters/postgres.rb:555:in `check_database_errors'
Jul 24 16:34:42 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         from /usr/share/gems/gems/sequel-5.68.0/lib/sequel/adapters/postgres.rb:487:in `listen'
Jul 24 16:34:42 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         from /usr/share/gems/gems/dynflow-1.7.0/lib/dynflow/connectors/database.rb:25:in `block in start'
Jul 24 16:34:42 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         from /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 24 16:34:50 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]: E, [2023-07-24T16:34:50.396900 #2715035] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Jul 24 16:34:50 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         This probably means the server terminated abnormally
Jul 24 16:34:50 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         before or while processing the request.
Jul 24 16:35:20 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]: E, [2023-07-24T16:35:20.399188 #2715035] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Jul 24 16:35:20 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         This probably means the server terminated abnormally
Jul 24 16:35:20 katello.server dynflow-sidekiq@worker-hosts-queue-1[2715035]:         before or while processing the request.
\ displaying foreman-proxy                                                      
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/foreman-proxy.service.d
           └─90-limits.conf
   Active: active (running) since Mon 2023-07-24 14:37:29 CEST; 19h ago
 Main PID: 2716093 (smart-proxy)
    Tasks: 8 (limit: 255042)
   Memory: 119.9M
   CGroup: /system.slice/foreman-proxy.service
           └─2716093 /usr/bin/ruby /usr/share/foreman-proxy/bin/smart-proxy

Jul 24 14:37:19 katello.server systemd[1]: Starting Foreman Proxy...
Jul 24 14:37:29 katello.server systemd[1]: Started Foreman Proxy.
\ All services are running                                            [OK]      
--------------------------------------------------------------------------------
1 Like
6

Pulp worker services look normal.

I am out of ideas, hoping someone else can step in. @katello

1 Like
7

If you look at the sync task in the foreman tasks browser, and then go into the dynflow console, what action is it getting stuck at? We have one bug that slowed down syncing that we’re addressing soon with 4.9.1, not sure if you’re hitting that.

1 Like
8

It was stuck on the 6: Actions::Pulp3::Repository::Sync Action

1 Like
9

@Kreggenwinkel As @iballou suggested, that does sound like you are experiencing: Katello 4.9 is very slow

Your options are to wait for 4.9.1: Katello 4.9.1 release process

Or to apply the patch/workaround from the thread I linked to.

2 Likes
10

Looks like it, thanks for the information!

1 Like
11

It might be worth testing that patch, if it doesn’t you might be hitting a different bug. Either way, if the patch or 4.9.1 doesn’t fix it, please let us know.

1 Like