Problem:
Create a new product and new private repo, unable to sync content
Expected outcome:
The expected outcome is for the content to be able to sync
Foreman and Proxy versions:
Foreman 3.11/Katello4.13
Foreman and Proxy plugin versions:
Simple repo mode, no special plugins at the moment
Distribution and version:
RHEL 8.10
Other relevant data:
Just create a new product and added a new internal repo, I tried to sync the content and is not completing succesfully.
It is showing the following outcome:
Pulp task error
@jeremylenz /@sajha , any pointers on this one?
sajha
June 27, 2024, 8:01pm
2
Hi @cucu007 ,
Is this a new repo URL? Would you be able to verify the foreman server has access to the repo url.
We can check further in /var/log/messages output when you run the sync if there’s any useful error messages.
Hi @sajha ,
Below is what I saw under /var/log/messages:
Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:49 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 10244 “-” “rest-client/2.1.0 (linux x86_64) ruby/2.7.8p225”
Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Starting task 01905c24-1f80-7bfc-8d76-35f59f1c51df
Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Task completed 01905c24-1f80-7bfc-8d76-35f59f1c51df
Jun 27 16:59:49 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:49 +0000] “PATCH /pulp/api/v3/remotes/rpm/rpm/01905b36-87e5-798c-8f61-dc29bdc42b27/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/3.25.3/ruby”
Jun 27 16:59:49 foreman.local pulpcore-api[367405]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:49 +0000] “GET /pulp/api/v3/tasks/01905c24-1f80-7bfc-8d76-35f59f1c51df/ HTTP/1.1” 200 750 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:50 foreman.local pulpcore-api[367414]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:50 +0000] “POST /pulp/api/v3/repositories/rpm/rpm/01905b36-8a21-77d5-bba3-7ac85b2e5f6b/sync/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/3.25.3/ruby”
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Starting task 01905c24-213a-74de-8b43-aa1826cae994
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo-rpms-16252963
Jun 27 16:59:50 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:50 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 841 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:50 foreman.local pulpcore-api[367409]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:50 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1242 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:51 foreman.local pulpcore-api[367405]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:51 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1474 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:52 foreman.local kernel: pulpcore-worker[444511]: segfault at 18 ip 00007f47360ae1fb sp 00007ffeab40c4e0 error 4 in _libpycomps.so[7f473609f000+1c000]
Jun 27 16:59:52 foreman.local kernel: Code: 7b 40 48 85 ff 74 12 48 c7 43 40 00 00 00 00 48 83 2f 01 0f 84 16 01 00 00 48 8b 7b 10 e8 9d ef ff ff 48 8b 45 08 48 89 43 10 <48> 8b 78 18 e8 8c ef ff ff 48 8b 43 10 48 8b 55 30 48 89 ef 48 89
Jun 27 16:59:52 foreman.local systemd[1]: Started Process Core Dump (PID 444522/UID 0).
Jun 27 16:59:52 foreman.local pulpcore-api[367409]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:52 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1477 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:52 foreman.local systemd-coredump[444523]: Resource limits disable core dumping for process 444511 (pulpcore-worker).
Jun 27 16:59:52 foreman.local systemd-coredump[444523]: Process 444511 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c24-213a-74de-8b43-aa1826cae994 exited with non zero exitcode -11.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c24-213a-74de-8b43-aa1826cae994 and marking as failed. Reason: Killed by signal 11.
Jun 27 16:59:52 foreman.local systemd[1]: systemd-coredump@9-444522-0.service: Succeeded.
Jun 27 16:59:52 foreman.local pulpcore-api[408769]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:52 +0000] “GET /pulp/api/v3/tasks/01905c24-213a-74de-8b43-aa1826cae994/ HTTP/1.1” 200 1526 “-” “OpenAPI-Generator/3.49.6/ruby”
Jun 27 16:59:53 foreman.local pulpcore-api[408769]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:53 +0000] “GET /pulp/api/v3/distributions/rpm/rpm/?base_path=MY_Organization%2FLibrary%2Fcustom%2FINTERNAL-EPEL-REPOS%2Fepel-internal-repo-rpms HTTP/1.1” 200 749 “-” “OpenAPI-Generator/3.25.3/ruby”
Jun 27 16:59:53 foreman.local pulpcore-api[408522]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: - - [27/Jun/2024:23:59:53 +0000] “GET /pulp/api/v3/publications/rpm/rpm/01905b36-8ec0-7d56-91ab-9107d33f784d/ HTTP/1.1” 200 569 “-” “OpenAPI-Generator/3.25.3/ruby”
sajha
June 28, 2024, 2:15am
4
Looks like you’re hitting a segfault from pulp and a resource limit of some kind. What is memory on the server you’re running this on? How many pulp workers do you see with sudo systemctl status pulpcore-worker*
?
Hi @sajha , the system has 64GB, this is the output of “free -g”
Below find the out put of the “systemctl status pulpcore-worker*”
systemctl status pulcore-worker
Unit pulcore-worker.service could not be found.
[root@foreman.local ~]# systemctl status pulcore-worker*
[root@foreman.local ~]# systemctl status pulpcore-worker*
● pulpcore-worker@8.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
Main PID: 367618 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 121.2M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@8.service
└─367618 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]: self.raise_for_status(response)
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]: File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]: response.raise_for_status()
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]: File "/usr/lib64/python3.11/site-packages/aiohttp/client_reqrep.py", line 1058, in raise_for_status
Jun 27 12:45:19 foreman.local pulpcore-worker-8[418981]: raise ClientResponseError(
Jun 27 12:46:13 foreman.local pulpcore-worker-8[419096]: pulp [bc8e27f1-7ae1-4438-aee2-4e29a2c25c68]: pulpcore.tasking.tasks:INFO: Starting task 01905b3b-f1bd-71e9-af46-ada409bc636a
Jun 27 12:46:13 foreman.local pulpcore-worker-8[419096]: pulp [bc8e27f1-7ae1-4438-aee2-4e29a2c25c68]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo-rpms>
Jun 27 12:46:15 foreman.local systemd-coredump[419107]: Process 419096 (pulpcore-worker) of user 989 dumped core.
Jun 27 12:46:15 foreman.local pulpcore-worker-8[367618]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905b3b-f1bd-71e9-af46-ada409bc636a exited with non zero exitcode -11.
Jun 27 12:46:15 foreman.local pulpcore-worker-8[367618]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905b3b-f1bd-71e9-af46-ada409bc636a and marking as failed. Reason: Killed by signal 11.
● pulpcore-worker@3.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
Main PID: 367622 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 120.9M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@3.service
└─367622 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 11:44:32 foreman.local pulpcore-worker-3[408275]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulp_rpm.app.tasks.publishing:INFO: Publication: 01905b03-7733-7a04-8a6a-87dafd0c86f4 created
Jun 27 11:44:32 foreman.local pulpcore-worker-3[408275]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulpcore.tasking.tasks:INFO: Task completed 01905b03-765b-70a6-b52c-38ee7b878a8d
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Starting task 01905b2b-2c90-768c-b61b-0f02080ab53f
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=epel-internal-repo-rpms-16249895, version=0
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulp_rpm.app.tasks.publishing:INFO: Publication: 01905b2b-2d57-7cd8-bbe4-cc1ef3ca5534 created
Jun 27 12:27:54 foreman.local pulpcore-worker-3[412939]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Task completed 01905b2b-2c90-768c-b61b-0f02080ab53f
Jun 27 12:27:55 foreman.local pulpcore-worker-3[412955]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Starting task 01905b2b-317e-7d0b-b227-376563b44a6f
Jun 27 12:27:56 foreman.local pulpcore-worker-3[412955]: pulp [70e60361-3198-4687-badf-49d0e0e16466]: pulpcore.tasking.tasks:INFO: Task completed 01905b2b-317e-7d0b-b227-376563b44a6f
Jun 27 12:38:25 foreman.local pulpcore-worker-3[418025]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Starting task 01905b34-cd1b-7c44-946a-c571067d897c
Jun 27 12:38:25 foreman.local pulpcore-worker-3[418025]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Task completed 01905b34-cd1b-7c44-946a-c571067d897c
● pulpcore-worker@7.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
Main PID: 367626 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 120.9M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@7.service
└─367626 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: File "/usr/lib/python3.11/site-packages/pulpcore/download/http.py", line 254, in download_wrapper
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: return await self._run(extra_data=extra_data)
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: self.raise_for_status(response)
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: response.raise_for_status()
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: File "/usr/lib64/python3.11/site-packages/aiohttp/client_reqrep.py", line 1058, in raise_for_status
Jun 27 12:28:15 foreman.local pulpcore-worker-7[413127]: raise ClientResponseError(
● pulpcore-worker@6.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:43 PDT; 22h ago
Main PID: 367459 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 121.2M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@6.service
└─367459 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 08:56:28 foreman.local pulpcore-worker-6[367459]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905a67-2612-79c3-adee-6ae016d4628f and marking as canceled.
Jun 27 11:44:33 foreman.local pulpcore-worker-6[408299]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulpcore.tasking.tasks:INFO: Starting task 01905b03-7b90-7d57-a911-1a42e864a1c8
Jun 27 11:44:33 foreman.local pulpcore-worker-6[408299]: pulp [0c19b4b6-704e-4322-8735-389da7e3bccd]: pulpcore.tasking.tasks:INFO: Task completed 01905b03-7b90-7d57-a911-1a42e864a1c8
Jun 27 11:44:40 foreman.local pulpcore-worker-6[408327]: pulp [776df382-a478-4e88-a48c-a6fc6f665776]: pulpcore.tasking.tasks:INFO: Starting task 01905b03-984b-7689-8c2b-97ba835a7a7a
Jun 27 11:44:40 foreman.local pulpcore-worker-6[408327]: pulp [776df382-a478-4e88-a48c-a6fc6f665776]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=G-rhel-8-for-x86_64-highavailability-e4s-rhui-rpms-16238515 remote=G-rhe>
Jun 27 11:45:08 foreman.local systemd-coredump[408479]: Process 408327 (pulpcore-worker) of user 989 dumped core.
Jun 27 11:45:08 foreman.local pulpcore-worker-6[367459]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905b03-984b-7689-8c2b-97ba835a7a7a exited with non zero exitcode -11.
Jun 27 11:45:08 foreman.local pulpcore-worker-6[367459]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905b03-984b-7689-8c2b-97ba835a7a7a and marking as failed. Reason: Killed by signal 11.
Jun 27 12:26:11 foreman.local pulpcore-worker-6[412773]: pulp [b0ee2563-46fb-40a3-a11d-a6303c66c9dc]: pulpcore.tasking.tasks:INFO: Starting task 01905b29-9a18-7202-ac86-e81563b3ae47
Jun 27 12:26:11 foreman.local pulpcore-worker-6[412773]: pulp [b0ee2563-46fb-40a3-a11d-a6303c66c9dc]: pulpcore.tasking.tasks:INFO: Task completed 01905b29-9a18-7202-ac86-e81563b3ae47
● pulpcore-worker@1.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
Main PID: 367621 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 120.9M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
└─367621 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: File "/usr/lib/python3.11/site-packages/pulpcore/download/http.py", line 254, in download_wrapper
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: return await self._run(extra_data=extra_data)
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: self.raise_for_status(response)
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: response.raise_for_status()
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: File "/usr/lib64/python3.11/site-packages/aiohttp/client_reqrep.py", line 1058, in raise_for_status
Jun 27 12:40:27 foreman.local pulpcore-worker-1[418469]: raise ClientResponseError(
● pulpcore-worker@5.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:43 PDT; 22h ago
Main PID: 367455 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 121.2M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@5.service
└─367455 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 06:46:43 foreman.local systemd[1]: Started Pulp Worker.
Jun 27 06:46:51 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Jun 27 06:46:51 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:INFO: New worker '367455@foreman.local' discovered
Jun 27 12:38:24 foreman.local pulpcore-worker-5[418015]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Starting task 01905b34-c77f-7574-9504-e749bab9f360
Jun 27 12:38:24 foreman.local pulpcore-worker-5[418015]: pulp [eae9cd78-199a-48d2-bfbb-70bb8580f806]: pulpcore.tasking.tasks:INFO: Task completed 01905b34-c77f-7574-9504-e749bab9f360
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulpcore.tasking.tasks:INFO: Starting task 01905c24-213a-74de-8b43-aa1826cae994
Jun 27 16:59:50 foreman.local pulpcore-worker-5[444511]: pulp [aa02c26d-1093-4273-a399-eb70c4a62843]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo>
Jun 27 16:59:52 foreman.local systemd-coredump[444523]: Process 444511 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c24-213a-74de-8b43-aa1826cae994 exited with non zero exitcode -11.
Jun 27 16:59:52 foreman.local pulpcore-worker-5[367455]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c24-213a-74de-8b43-aa1826cae994 and marking as failed. Reason: Killed by signal 11.
● pulpcore-worker@2.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:43 PDT; 22h ago
Main PID: 367450 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 121.1M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
└─367450 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 11:36:54 foreman.local pulpcore-worker-2[407267]: pulp [c2f65350-d6ad-4600-a420-54eafcac3b8f]: pulpcore.tasking.tasks:INFO: Starting task 01905afc-7a81-7cfb-b820-1ff8ad311c0d
Jun 27 11:36:54 foreman.local pulpcore-worker-2[407267]: pulp [c2f65350-d6ad-4600-a420-54eafcac3b8f]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=G-ansible-2-for-rhel-8-rhui-rpms-1622380 remote=G-ansible-2-for-rhel-8-r>
Jun 27 11:36:56 foreman.local systemd-coredump[407284]: Process 407267 (pulpcore-worker) of user 989 dumped core.
Jun 27 11:36:56 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905afc-7a81-7cfb-b820-1ff8ad311c0d exited with non zero exitcode -11.
Jun 27 11:36:56 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905afc-7a81-7cfb-b820-1ff8ad311c0d and marking as failed. Reason: Killed by signal 11.
Jun 27 16:58:48 foreman.local pulpcore-worker-2[444397]: pulp [a4b86d5d-26ea-43ad-9330-ba019a8ef8c9]: pulpcore.tasking.tasks:INFO: Starting task 01905c23-2f26-79b4-ba5f-dfa4111dfd4b
Jun 27 16:58:48 foreman.local pulpcore-worker-2[444397]: pulp [a4b86d5d-26ea-43ad-9330-ba019a8ef8c9]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo>
Jun 27 16:58:50 foreman.local systemd-coredump[444412]: Process 444397 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:58:50 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c23-2f26-79b4-ba5f-dfa4111dfd4b exited with non zero exitcode -11.
Jun 27 16:58:50 foreman.local pulpcore-worker-2[367450]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c23-2f26-79b4-ba5f-dfa4111dfd4b and marking as failed. Reason: Killed by signal 11.
● pulpcore-worker@4.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-27 06:46:44 PDT; 22h ago
Main PID: 367624 (pulpcore-worker)
Tasks: 1 (limit: 411736)
Memory: 121.1M
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
└─367624 /usr/bin/python3.11 /usr/bin/pulpcore-worker
Jun 27 11:47:35 foreman.local pulpcore-worker-4[367624]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905b05-e3af-7d90-a755-4dbb61d4e67e and marking as failed. Reason: Killed by signal 11.
Jun 27 12:27:00 foreman.local pulpcore-worker-4[412870]: pulp [96ea9668-6f08-4521-a478-f3a7d1af6d2a]: pulpcore.tasking.tasks:INFO: Starting task 01905b2a-5a48-757d-a5d5-475170e78bfd
Jun 27 12:27:01 foreman.local pulpcore-worker-4[412870]: pulp [96ea9668-6f08-4521-a478-f3a7d1af6d2a]: pulpcore.tasking.tasks:INFO: Task completed 01905b2a-5a48-757d-a5d5-475170e78bfd
Jun 27 12:40:21 foreman.local pulpcore-worker-4[418419]: pulp [81b2a498-bd12-470d-bdfc-c25bb5b8400f]: pulpcore.tasking.tasks:INFO: Starting task 01905b36-92da-7897-b902-08dea655a910
Jun 27 12:40:21 foreman.local pulpcore-worker-4[418419]: pulp [81b2a498-bd12-470d-bdfc-c25bb5b8400f]: pulpcore.tasking.tasks:INFO: Task completed 01905b36-92da-7897-b902-08dea655a910
Jun 27 16:56:36 foreman.local pulpcore-worker-4[444212]: pulp [7c0a8c6e-c676-49da-9e3c-9c134932c6bf]: pulpcore.tasking.tasks:INFO: Starting task 01905c21-2d0f-718c-99de-095f61094d05
Jun 27 16:56:36 foreman.local pulpcore-worker-4[444212]: pulp [7c0a8c6e-c676-49da-9e3c-9c134932c6bf]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=epel-internal-repo-rpms-16253137 remote=epel-internal-repo>
Jun 27 16:56:38 foreman.local systemd-coredump[444223]: Process 444212 (pulpcore-worker) of user 989 dumped core.
Jun 27 16:56:38 foreman.local pulpcore-worker-4[367624]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01905c21-2d0f-718c-99de-095f61094d05 exited with non zero exitcode -11.
Jun 27 16:56:38 foreman.local pulpcore-worker-4[367624]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 01905c21-2d0f-718c-99de-095f61094d05 and marking as failed. Reason: Killed by signal 11.
sajha
June 28, 2024, 3:01pm
6
That seems right to me… @dralley : Do you know why the sync might be failing?
From the mere fact that it coredumped? No. Can you inspect the coredump file and share a traceback from within that?
Hi @sajha /@dralley , please let me know how I can collect the coredump element for review…I will be happy to collect and share accordingly. Thank you.
Take a look at the coredumpctl
tool. List the coredumps, see if you can find the correct one (likely the latest one) and paste the results of coredumpctl info
for that dump
dralley:
coredumpctl
Hi @dralley ,
Let me know if the below shines any light:
coredumpctl info 538553
PID: 538553 (pulpcore-worker)
UID: 989 (pulp)
GID: 986 (pulp)
Signal: 11 (SEGV)
Timestamp: Fri 2024-06-28 05:54:06 PDT (3h 40min ago)
Command Line: /usr/bin/python3.11 /usr/bin/pulpcore-worker
Executable: /usr/bin/python3.11
Control Group: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
Unit: pulpcore-worker@4.service
Slice: system-pulpcore\x2dworker.slice
Boot ID: 60347a3c5dd94b44920d25e0a0470c4a
Machine ID: e47d5098cf034d78b4dd6c57c6553aa1
Hostname: foreman.local
Storage: none
Message: Process 538553 (pulpcore-worker) of user 989 dumped core.
@dralley , below is the entire list of the coredumpctl:
above I just grab the latest one.
coredumpctl list
TIME PID UID GID SIG COREFILE EXE
Thu 2024-06-27 11:25:18 PDT 405839 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 11:31:58 PDT 406676 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 11:36:56 PDT 407267 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 11:40:23 PDT 407769 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 11:45:08 PDT 408327 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 11:47:35 PDT 408736 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 12:46:15 PDT 419096 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 16:56:38 PDT 444212 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 16:58:50 PDT 444397 989 986 11 none /usr/bin/python3.11
Thu 2024-06-27 16:59:52 PDT 444511 989 986 11 none /usr/bin/python3.11
Fri 2024-06-28 05:20:10 PDT 527387 989 986 11 none /usr/bin/python3.11
Fri 2024-06-28 05:21:00 PDT 527573 989 986 11 none /usr/bin/python3.11
Fri 2024-06-28 05:31:40 PDT 530811 989 986 11 none /usr/bin/python3.11
Fri 2024-06-28 05:46:34 PDT 537549 989 986 11 none /usr/bin/python3.11
Fri 2024-06-28 05:54:06 PDT 538553 989 986 11 none /usr/bin/python3.11
Is that all it provides, no traceback? Normally there would be a short traceback at the bottom.
Could you try dumping it with coredumpctl dump
and uploading the file?
Hi @dralley , what command would I need to execute, sorry about my ignorance but tracedump is new to me.
It’s been a while since I’ve done anything like this, my knowledge is also a bit rusty.
Try coredumpctl gdb 538553
and then once you’re in a shell, type
bt
and see if you can copy and paste the output. I hope I’m not missing steps.
dralley:
coredumpctl gdb 538553
Below find the output:
coredumpctl gdb 538553
PID: 538553 (pulpcore-worker)
UID: 989 (pulp)
GID: 986 (pulp)
Signal: 11 (SEGV)
Timestamp: Fri 2024-06-28 05:54:06 PDT (6h ago)
Command Line: /usr/bin/python3.11 /usr/bin/pulpcore-worker
Executable: /usr/bin/python3.11
Control Group: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
Unit: pulpcore-worker@4.service
Slice: system-pulpcore\x2dworker.slice
Boot ID: 60347a3c5dd94b44920d25e0a0470c4a
Machine ID: e47d5098cf034d78b4dd6c57c6553aa1
Hostname: foreman.local
Storage: none
Message: Process 538553 (pulpcore-worker) of user 989 dumped core.
Coredump entry has no core attached (neither internally in the journal nor externally on disk).
@dralley . bt is returning “-bash: bt: command not found”
It sounds like you’re running it in bash, not inside the gdb shell
Hi @dralley , false alert…the repo sync is happening as normal.
I do have a quick question, when I am running a reposyn I normally go to the “sync status” and are able to see the active tasks by checking the box. The status is no longer showing when the task runs, any clues if thats a bug?
sajha
July 1, 2024, 2:23pm
19
We have a fix in upstream for the sync status not refreshing on the page. Fixes #37442 - Sync status page presenter doesn't work by sjha4 · Pull Request #10993 · Katello/katello · GitHub
It will be back-ported up to release 4.12 and newer when the next releases come out.
Hi @sajha ,
How do I apply the fix? Do you have some steps for those of us non-programmers?