Foreman 3.11/Katello4.13 - Unable to sync private repository

cucu007 · June 27, 2024, 7:50pm

Problem:
Create a new product and new private repo, unable to sync content
Expected outcome:
The expected outcome is for the content to be able to sync
Foreman and Proxy versions:
Foreman 3.11/Katello4.13
Foreman and Proxy plugin versions:
Simple repo mode, no special plugins at the moment
Distribution and version:
RHEL 8.10
Other relevant data:
Just create a new product and added a new internal repo, I tried to sync the content and is not completing succesfully.

It is showing the following outcome:

Pulp task error

@jeremylenz /@sajha , any pointers on this one?

sajha · June 27, 2024, 8:01pm

Hi @cucu007 ,

Is this a new repo URL? Would you be able to verify the foreman server has access to the repo url.

We can check further in /var/log/messages output when you run the sync if there’s any useful error messages.

cucu007 · June 28, 2024, 12:05am

Hi @sajha ,
Below is what I saw under /var/log/messages:

Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:49 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 10244 “-” “rest-client/2.1.0 (linux x86_64) ruby/2.7.8p225”
Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Starting task 01905c24-1f80-7bfc-8d76-35f59f1c51df
Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Task completed 01905c24-1f80-7bfc-8d76-35f59f1c51df
Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:49 +0000] “PATCH /pulp/api/v3/remotes/rpm/rpm/01905b36-87e5-798c-8f61-dc29bdc42b27/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/3.25.3/ruby”
Jun 27 16:59:49 foreman.local pulpcore-api[367405]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:49 +0000] “GET /pulp/api/v3/tasks/01905c24-1f80-7bfc-8d76-35f59f1c51df/ HTTP/1.1” 200 750 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:50 foreman.local pulpcore-api[367414]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:50 +0000] “POST /pulp/api/v3/repositories/rpm/rpm/01905b36-8a21-77d5-bba3-7ac85b2e5f6b/sync/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/3.25.3/ruby”
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Starting task 01905c24-213a-74de-8b43-aa1826cae994
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo-rpms-16252963
Jun 27 16:59:50 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:50 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 841 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:50 foreman.local pulpcore-api[367409]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:50 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1242 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:51 foreman.local pulpcore-api[367405]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:51 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1474 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:52 foreman.local kernel: pulpcore-worker[444511]: segfault at 18 ip 00007f47360ae1fb sp 00007ffeab40c4e0 error 4 in _libpycomps.so[7f473609f000+1c000]
Jun 27 16:59:52 foreman.local kernel: Code: 7b 40 48 85 ff 74 12 48 c7 43 40 00 00 00 00 48 83 2f 01 0f 84 16 01 00 00 48 8b 7b 10 e8 9d ef ff ff 48 8b 45 08 48 89 43 10 <48> 8b 78 18 e8 8c ef ff ff 48 8b 43 10 48 8b 55 30 48 89 ef 48 89
Jun 27 16:59:52 foreman.local systemd[1]: Started Process Core Dump (PID 444522/UID 0).
Jun 27 16:59:52 foreman.local pulpcore-api[367409]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:52 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1477 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:52 foreman.local systemd-coredump[444523]: Resource limits disable core dumping for process 444511 (pulpcore-worker).
Jun 27 16:59:52 foreman.local systemd-coredump[444523]: Process 444511 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c24-213a-74de-8b43-aa1826cae994 exited with non zero exitcode -11.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c24-213a-74de-8b43-aa1826cae994 and marking as failed. Reason: Killed by signal 11.
Jun 27 16:59:52 foreman.local systemd[1]: systemd-coredump@9-444522-0.service: Succeeded.
Jun 27 16:59:52 foreman.local pulpcore-api[408769]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:52 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1526 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:53 foreman.local pulpcore-api[408769]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:53 +0000] “GET /pulp/api/v3/distributions/rpm/rpm/?base_path=MY_Organization%2FLibrary%2Fcustom%2FINTERNAL-EPEL-REPOS%2Fepel-internal-repo-rpms HTTP/1.1” 200 749 “-” “OpenAPI-Generator/3.25.3/ruby”
Jun 27 16:59:53 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:53 +0000] “GET /pulp/api/v3/publications/rpm/rpm/01905b36-8ec0-7d56-91ab-9107d33f784d/ HTTP/1.1” 200 569 “-” “OpenAPI-Generator/3.25.3/ruby”

sajha · June 28, 2024, 2:15am

Looks like you’re hitting a segfault from pulp and a resource limit of some kind. What is memory on the server you’re running this on? How many pulp workers do you see with sudo systemctl status pulpcore-worker* ?

cucu007 · June 28, 2024, 11:54am

Hi @sajha , the system has 64GB, this is the output of “free -g”

Below find the out put of the “systemctl status pulpcore-worker*”

systemctl status pulcore-worker
Unit pulcore-worker.service could not be found.
[root@foreman.local ~]# systemctl status pulcore-worker*
[root@foreman.local ~]# systemctl status pulpcore-worker*
● pulpcore-worker@8.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
 Main PID: 367618 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 121.2M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@8.service
           └─367618 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]:     self.raise_for_status(response)
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]:   File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]:     response.raise_for_status()
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]:   File "/usr/lib64/python3.11/site-packages/aiohttp/client_reqrep.py", line 1058, in raise_for_status
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]:     raise ClientResponseError(
Jun 27 12:46:13 foreman.local pulpcore-worker-8[419096]: pulp [bc8e27f1-7ae1-4438-aee2-4e29a2c25c68]: pulpcore.tasking.tasks:INFO: Starting task 01905b3b-f1bd-71e9-af46-ada409bc636a
Jun 27 12:46:13 foreman.local pulpcore-worker-8[419096]: pulp [bc8e27f1-7ae1-4438-aee2-4e29a2c25c68]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo-rpms>
Jun 27 12:46:15 foreman.local systemd-coredump[419107]: Process 419096 (pulpcore-worker) of user 989 dumped core.
Jun 27 12:46:15 foreman.local pulpcore-worker-8[367618]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905b3b-f1bd-71e9-af46-ada409bc636a exited with non zero exitcode -11.
Jun 27 12:46:15 foreman.local pulpcore-worker-8[367618]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905b3b-f1bd-71e9-af46-ada409bc636a and marking as failed. Reason: Killed by signal 11.

● pulpcore-worker@3.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
 Main PID: 367622 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 120.9M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@3.service
           └─367622 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 11:44:32 foreman.local pulpcore-worker-3[408275]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulp_rpm.app.tasks.publishing:INFO: Publication: 01905b03-7733-7a04-8a6a-87dafd0c86f4 created
Jun 27 11:44:32 foreman.local pulpcore-worker-3[408275]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulpcore.tasking.tasks:INFO: Task completed 01905b03-765b-70a6-b52c-38ee7b878a8d
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Starting task 01905b2b-2c90-768c-b61b-0f02080ab53f
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=epel-internal-repo-rpms-16249895, version=0
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulp_rpm.app.tasks.publishing:INFO: Publication: 01905b2b-2d57-7cd8-bbe4-cc1ef3ca5534 created
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Task completed 01905b2b-2c90-768c-b61b-0f02080ab53f
Jun 27 12:27:55 foreman.local pulpcore-worker-3[412955]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Starting task 01905b2b-317e-7d0b-b227-376563b44a6f
Jun 27 12:27:56 foreman.local pulpcore-worker-3[412955]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Task completed 01905b2b-317e-7d0b-b227-376563b44a6f
Jun 27 12:38:25 foreman.local pulpcore-worker-3[418025]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Starting task 01905b34-cd1b-7c44-946a-c571067d897c
Jun 27 12:38:25 foreman.local pulpcore-worker-3[418025]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Task completed 01905b34-cd1b-7c44-946a-c571067d897c

● pulpcore-worker@7.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
 Main PID: 367626 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 120.9M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@7.service
           └─367626 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:   File "/usr/lib/python3.11/site-packages/pulpcore/download/http.py", line 254, in download_wrapper
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:     return await self._run(extra_data=extra_data)
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:   File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:     self.raise_for_status(response)
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:   File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:     response.raise_for_status()
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:   File "/usr/lib64/python3.11/site-packages/aiohttp/client_reqrep.py", line 1058, in raise_for_status
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]:     raise ClientResponseError(

● pulpcore-worker@6.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:43 PDT; 22h ago
 Main PID: 367459 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 121.2M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@6.service
           └─367459 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 08:56:28 foreman.local pulpcore-worker-6[367459]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905a67-2612-79c3-adee-6ae016d4628f and marking as canceled.
Jun 27 11:44:33 foreman.local pulpcore-worker-6[408299]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulpcore.tasking.tasks:INFO: Starting task 01905b03-7b90-7d57-a911-1a42e864a1c8
Jun 27 11:44:33 foreman.local pulpcore-worker-6[408299]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulpcore.tasking.tasks:INFO: Task completed 01905b03-7b90-7d57-a911-1a42e864a1c8
Jun 27 11:44:40 foreman.local pulpcore-worker-6[408327]: pulp [776df382-a478-4e88-a48c-a6fc6f665776]: pulpcore.tasking.tasks:INFO: Starting task 01905b03-984b-7689-8c2b-97ba835a7a7a
Jun 27 11:44:40 foreman.local pulpcore-worker-6[408327]: pulp [776df382-a478-4e88-a48c-a6fc6f665776]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=G-rhel-8-for-x86_64-highavailability-e4s-rhui-rpms-16238515 remote=G-rhe>
Jun 27 11:45:08 foreman.local systemd-coredump[408479]: Process 408327 (pulpcore-worker) of user 989 dumped core.
Jun 27 11:45:08 foreman.local pulpcore-worker-6[367459]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905b03-984b-7689-8c2b-97ba835a7a7a exited with non zero exitcode -11.
Jun 27 11:45:08 foreman.local pulpcore-worker-6[367459]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905b03-984b-7689-8c2b-97ba835a7a7a and marking as failed. Reason: Killed by signal 11.
Jun 27 12:26:11 foreman.local pulpcore-worker-6[412773]: pulp [b0ee2563-46fb-40a3-a11d-a6303c66c9dc]: pulpcore.tasking.tasks:INFO: Starting task 01905b29-9a18-7202-ac86-e81563b3ae47
Jun 27 12:26:11 foreman.local pulpcore-worker-6[412773]: pulp [b0ee2563-46fb-40a3-a11d-a6303c66c9dc]: pulpcore.tasking.tasks:INFO: Task completed 01905b29-9a18-7202-ac86-e81563b3ae47

● pulpcore-worker@1.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
 Main PID: 367621 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 120.9M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
           └─367621 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:   File "/usr/lib/python3.11/site-packages/pulpcore/download/http.py", line 254, in download_wrapper
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:     return await self._run(extra_data=extra_data)
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:   File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:     self.raise_for_status(response)
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:   File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:     response.raise_for_status()
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:   File "/usr/lib64/python3.11/site-packages/aiohttp/client_reqrep.py", line 1058, in raise_for_status
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]:     raise ClientResponseError(

● pulpcore-worker@5.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:43 PDT; 22h ago
 Main PID: 367455 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 121.2M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@5.service
           └─367455 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 06:46:43 foreman.local systemd[1]: Started Pulp Worker.
Jun 27 06:46:51 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Jun 27 06:46:51 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:INFO: New worker '367455@foreman.local' discovered
Jun 27 12:38:24 foreman.local pulpcore-worker-5[418015]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Starting task 01905b34-c77f-7574-9504-e749bab9f360
Jun 27 12:38:24 foreman.local pulpcore-worker-5[418015]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Task completed 01905b34-c77f-7574-9504-e749bab9f360
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Starting task 01905c24-213a-74de-8b43-aa1826cae994
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo>
Jun 27 16:59:52 foreman.local systemd-coredump[444523]: Process 444511 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c24-213a-74de-8b43-aa1826cae994 exited with non zero exitcode -11.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c24-213a-74de-8b43-aa1826cae994 and marking as failed. Reason: Killed by signal 11.

● pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:43 PDT; 22h ago
 Main PID: 367450 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 121.1M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
           └─367450 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 11:36:54 foreman.local pulpcore-worker-2[407267]: pulp [c2f65350-d6ad-4600-a420-54eafcac3b8f]: pulpcore.tasking.tasks:INFO: Starting task 01905afc-7a81-7cfb-b820-1ff8ad311c0d
Jun 27 11:36:54 foreman.local pulpcore-worker-2[407267]: pulp [c2f65350-d6ad-4600-a420-54eafcac3b8f]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=G-ansible-2-for-rhel-8-rhui-rpms-1622380 remote=G-ansible-2-for-rhel-8-r>
Jun 27 11:36:56 foreman.local systemd-coredump[407284]: Process 407267 (pulpcore-worker) of user 989 dumped core.
Jun 27 11:36:56 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905afc-7a81-7cfb-b820-1ff8ad311c0d exited with non zero exitcode -11.
Jun 27 11:36:56 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905afc-7a81-7cfb-b820-1ff8ad311c0d and marking as failed. Reason: Killed by signal 11.
Jun 27 16:58:48 foreman.local pulpcore-worker-2[444397]: pulp [a4b86d5d-26ea-43ad-9330-ba019a8ef8c9]: pulpcore.tasking.tasks:INFO: Starting task 01905c23-2f26-79b4-ba5f-dfa4111dfd4b
Jun 27 16:58:48 foreman.local pulpcore-worker-2[444397]: pulp [a4b86d5d-26ea-43ad-9330-ba019a8ef8c9]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo>
Jun 27 16:58:50 foreman.local systemd-coredump[444412]: Process 444397 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:58:50 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c23-2f26-79b4-ba5f-dfa4111dfd4b exited with non zero exitcode -11.
Jun 27 16:58:50 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c23-2f26-79b4-ba5f-dfa4111dfd4b and marking as failed. Reason: Killed by signal 11.

● pulpcore-worker@4.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
 Main PID: 367624 (pulpcore-worker)
    Tasks: 1 (limit: 411736)
   Memory: 121.1M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
           └─367624 /usr/bin/python3.11 /usr/bin/pulpcore-worker

Jun 27 11:47:35 foreman.local pulpcore-worker-4[367624]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905b05-e3af-7d90-a755-4dbb61d4e67e and marking as failed. Reason: Killed by signal 11.
Jun 27 12:27:00 foreman.local pulpcore-worker-4[412870]: pulp [96ea9668-6f08-4521-a478-f3a7d1af6d2a]: pulpcore.tasking.tasks:INFO: Starting task 01905b2a-5a48-757d-a5d5-475170e78bfd
Jun 27 12:27:01 foreman.local pulpcore-worker-4[412870]: pulp [96ea9668-6f08-4521-a478-f3a7d1af6d2a]: pulpcore.tasking.tasks:INFO: Task completed 01905b2a-5a48-757d-a5d5-475170e78bfd
Jun 27 12:40:21 foreman.local pulpcore-worker-4[418419]: pulp [81b2a498-bd12-470d-bdfc-c25bb5b8400f]: pulpcore.tasking.tasks:INFO: Starting task 01905b36-92da-7897-b902-08dea655a910
Jun 27 12:40:21 foreman.local pulpcore-worker-4[418419]: pulp [81b2a498-bd12-470d-bdfc-c25bb5b8400f]: pulpcore.tasking.tasks:INFO: Task completed 01905b36-92da-7897-b902-08dea655a910
Jun 27 16:56:36 foreman.local pulpcore-worker-4[444212]: pulp [7c0a8c6e-c676-49da-9e3c-9c134932c6bf]: pulpcore.tasking.tasks:INFO: Starting task 01905c21-2d0f-718c-99de-095f61094d05
Jun 27 16:56:36 foreman.local pulpcore-worker-4[444212]: pulp [7c0a8c6e-c676-49da-9e3c-9c134932c6bf]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo>
Jun 27 16:56:38 foreman.local systemd-coredump[444223]: Process 444212 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:56:38 foreman.local pulpcore-worker-4[367624]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c21-2d0f-718c-99de-095f61094d05 exited with non zero exitcode -11.
Jun 27 16:56:38 foreman.local pulpcore-worker-4[367624]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c21-2d0f-718c-99de-095f61094d05 and marking as failed. Reason: Killed by signal 11.

sajha · June 28, 2024, 3:01pm

That seems right to me… @dralley : Do you know why the sync might be failing?

dralley · June 28, 2024, 3:12pm

From the mere fact that it coredumped? No. Can you inspect the coredump file and share a traceback from within that?

cucu007 · June 28, 2024, 4:02pm

Hi @sajha /@dralley , please let me know how I can collect the coredump element for review…I will be happy to collect and share accordingly. Thank you.

dralley · June 28, 2024, 4:27pm

Take a look at the coredumpctl tool. List the coredumps, see if you can find the correct one (likely the latest one) and paste the results of coredumpctl info for that dump

cucu007 · June 28, 2024, 4:36pm

Hi @dralley ,
Let me know if the below shines any light:

coredumpctl info 538553
           PID: 538553 (pulpcore-worker)
           UID: 989 (pulp)
           GID: 986 (pulp)
        Signal: 11 (SEGV)
     Timestamp: Fri 2024-06-28 05:54:06 PDT (3h 40min ago)
  Command Line: /usr/bin/python3.11 /usr/bin/pulpcore-worker
    Executable: /usr/bin/python3.11
 Control Group: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
          Unit: pulpcore-worker@4.service
         Slice: system-pulpcore\x2dworker.slice
       Boot ID: 60347a3c5dd94b44920d25e0a0470c4a
    Machine ID: e47d5098cf034d78b4dd6c57c6553aa1
      Hostname: foreman.local
       Storage: none
       Message: Process 538553 (pulpcore-worker) of user 989 dumped core.

cucu007 · June 28, 2024, 4:37pm

@dralley , below is the entire list of the coredumpctl:

above I just grab the latest one.

coredumpctl list
TIME                            PID   UID   GID SIG COREFILE  EXE
Thu 2024-06-27 11:25:18 PDT  405839   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 11:31:58 PDT  406676   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 11:36:56 PDT  407267   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 11:40:23 PDT  407769   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 11:45:08 PDT  408327   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 11:47:35 PDT  408736   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 12:46:15 PDT  419096   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 16:56:38 PDT  444212   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 16:58:50 PDT  444397   989   986  11 none      /usr/bin/python3.11
Thu 2024-06-27 16:59:52 PDT  444511   989   986  11 none      /usr/bin/python3.11
Fri 2024-06-28 05:20:10 PDT  527387   989   986  11 none      /usr/bin/python3.11
Fri 2024-06-28 05:21:00 PDT  527573   989   986  11 none      /usr/bin/python3.11
Fri 2024-06-28 05:31:40 PDT  530811   989   986  11 none      /usr/bin/python3.11
Fri 2024-06-28 05:46:34 PDT  537549   989   986  11 none      /usr/bin/python3.11
Fri 2024-06-28 05:54:06 PDT  538553   989   986  11 none      /usr/bin/python3.11

dralley · June 28, 2024, 4:51pm

Is that all it provides, no traceback? Normally there would be a short traceback at the bottom.

Could you try dumping it with coredumpctl dump and uploading the file?

cucu007 · June 28, 2024, 6:39pm

Hi @dralley , what command would I need to execute, sorry about my ignorance but tracedump is new to me.

dralley · June 28, 2024, 7:11pm

It’s been a while since I’ve done anything like this, my knowledge is also a bit rusty.

Try coredumpctl gdb 538553

and then once you’re in a shell, type

bt

and see if you can copy and paste the output. I hope I’m not missing steps.

cucu007 · June 28, 2024, 7:42pm

Below find the output:

coredumpctl gdb 538553
PID: 538553 (pulpcore-worker)
UID: 989 (pulp)
GID: 986 (pulp)
Signal: 11 (SEGV)
Timestamp: Fri 2024-06-28 05:54:06 PDT (6h ago)
Command Line: /usr/bin/python3.11 /usr/bin/pulpcore-worker
Executable: /usr/bin/python3.11
Control Group: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
Unit: pulpcore-worker@4.service
Slice: system-pulpcore\x2dworker.slice
Boot ID: 60347a3c5dd94b44920d25e0a0470c4a
Machine ID: e47d5098cf034d78b4dd6c57c6553aa1
Hostname: foreman.local
Storage: none
Message: Process 538553 (pulpcore-worker) of user 989 dumped core.

Coredump entry has no core attached (neither internally in the journal nor externally on disk).

cucu007 · June 28, 2024, 7:50pm

@dralley . bt is returning “-bash: bt: command not found”

dralley · June 30, 2024, 1:50am

It sounds like you’re running it in bash, not inside the gdb shell

cucu007 · June 30, 2024, 3:21pm

Hi @dralley , false alert…the repo sync is happening as normal.

I do have a quick question, when I am running a reposyn I normally go to the “sync status” and are able to see the active tasks by checking the box. The status is no longer showing when the task runs, any clues if thats a bug?

sajha · July 1, 2024, 2:23pm

We have a fix in upstream for the sync status not refreshing on the page. Fixes #37442 - Sync status page presenter doesn't work by sjha4 · Pull Request #10993 · Katello/katello · GitHub

It will be back-ported up to release 4.12 and newer when the next releases come out.

cucu007 · July 1, 2024, 8:03pm

Hi @sajha ,
How do I apply the fix? Do you have some steps for those of us non-programmers?