Help with proxy sync failure please

Support
1

Problem:
Proxy sync fails after content promotion
Expected outcome:
Success on all promotions
Foreman and Proxy versions:
3.15/ katello 4.17

Hi please could someone advise why I often get failed tasks for some proxy syncs (but not all)

Below is one error log that I really need help with. This is from the dynflow console but happy to provide more details if needed. This happens most months now - we have 4 smart proxies and 1 tends to fail during the proxy sync of various repo data. According to the logs, some repo ids sync fine, some fail. Below is an example fail:

Output:

smart_proxy_history_id: 22846
pulp_tasks:

  • pulp_href: “/pulp/api/v3/tasks/0199b8de-42f9-7cbf-b18b-5550263e3066/”
    prn: prn:core.task:0199b8de-42f9-7cbf-b18b-5550263e3066
    pulp_created: ‘2025-10-06T09:33:17.945Z’
    pulp_last_updated: ‘2025-10-06T09:33:17.945Z’
    state: failed
    name: pulp_rpm.app.tasks.synchronizing.synchronize
    logging_cid: ‘0783b25c-7887-44ac-b649-fe7c60188a2c’
    created_by: “/pulp/api/v3/users/1/”
    unblocked_at: ‘2025-10-06T09:33:17.973Z’
    started_at: ‘2025-10-06T09:41:16.244Z’
    finished_at: ‘2025-10-06T09:45:07.724Z’
    error:
    reason: Killed by signal 11.
    worker: “/pulp/api/v3/workers/0197c640-91ed-7ec0-b36c-c5a0ab7711e8/”
    child_tasks:
    progress_reports:
    • message: Downloading Metadata Files
      code: sync.downloading.metadata
      state: completed
      done: 10
      suffix:
    • message: Downloading Artifacts
      code: sync.downloading.artifacts
      state: running
      done: 0
      suffix:
    • message: Associating Content
      code: associating.content
      state: running
      done: 20
      suffix:
    • message: Skipping Packages
      code: sync.skipped.packages
      state: completed
      total: 0
      done: 0
      suffix:
    • message: Parsed Packages
      code: sync.parsing.packages
      state: completed
      total: 27308
      done: 27308
      suffix:
    • message: Parsed Comps
      code: sync.parsing.comps
      state: completed
      total: 42
      done: 42
      suffix:
      created_resources:
      reserved_resources_record:
    • prn:rpm.rpmrepository:018ec381-f4f5-7340-b0ff-286cf4e0e3cb
    • shared:prn:rpm.rpmremote:018ec381-f783-705a-9881-91aa480febeb
    • shared:prn:core.domain:018ebdd9-aa7d-7825-9c0d-6ac52ddecc63
      task_groups:
      poll_attempts:
      total: 50
      failed: 1
      Chunked output:

—
Error:

Katello::Errors::Pulp3Error

Pulp task error

  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:107:in
    `block in check_for_errors’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:105:in
    `each’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:105:in
    `check_for_errors’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:161:in
    `poll_external_task’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/polling.rb:22:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/cancellable.rb:15:in `run’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:590:in `block (3 levels)
    in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:26:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:33:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:16:in
    `block in run’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:40:in
    `block in as_remote_user’”
  • “/usr/share/gems/gems/katello-4.17.0/app/models/katello/concerns/user_extensions.rb:21:in
    `cp_config’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:27:in
    `as_cp_user’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:39:in
    `as_remote_user’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:16:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
    `block in run’”
  • “/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in
    `wrap’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/progress.rb:29:in `with_progress_calculation’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/progress.rb:15:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/load_setting_values.rb:20:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in
    `restore_current_request_id’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in
    `restore_curent_timezone’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
    `restore_current_taxonomies’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:33:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in
    `restore_curent_user’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/world.rb:31:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:589:in `block (2 levels)
    in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:588:in `catch’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:588:in `block in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:491:in `block in with_error_handling’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:491:in `catch’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:491:in `with_error_handling’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:583:in `execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:304:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
    `block (2 levels) in execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract.rb:168:in
    `with_meta_calculation’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
    `block in execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
    `open_action’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
    `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/director.rb:95:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in
    `block (2 levels) in perform’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors.rb:18:in `run_user_code’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in
    `block in perform’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in
    `with_telemetry’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
    `perform’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/serialization.rb:28:in
    `perform’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels)
    in process’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in module:Sidekiq’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread’”
  • “/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block
    in create_with_logging_context’”
2

Hi @chatlow ,

The Pulp3 error means Pulp likely threw a stack trace somewhere. If you take a look in /var/log/messages on your failing smart proxy, you should be able to find it there. Then we might be able to determine if there is a Pulp bug that’s being hit.

3

@iballou Struggled to find initially but managed to get these from messages on the proxy which show the exit code (just before and after), hope they help you

Oct 6 09:45:02 nj-prod-foreman-proxy01 kernel: traps: pulpcore-worker[785448] general protection fault ip:7fe7b5024117 sp:7ffc4fe2c270 error:0 in libpython3.12.so.1.0[7fe7b4efe000+276000]
Oct 6 09:45:02 nj-prod-foreman-proxy01 systemd[1]: Started Process Core Dump (PID 786026/UID 0).
Oct 6 09:45:03 nj-prod-foreman-proxy01 pulpcore-api[778772]: pulp [0783b25c-7887-44ac-b649-fe7c60188a2c]: - - [06/Oct/2025:09:45:03 +0000] “GET /pulp/api/v3/tasks/0199b8e8-b002-769a-be39-fd05b0a08f99/ HTTP/1.1” 200 715 “-” “OpenAPI-G
enerator/3.73.9/ruby”
Oct 6 09:45:06 nj-prod-foreman-proxy01 pulpcore-api[778772]: pulp [0783b25c-7887-44ac-b649-fe7c60188a2c]: - - [06/Oct/2025:09:45:06 +0000] “GET /pulp/api/v3/tasks/0199b8e8-b002-769a-be39-fd05b0a08f99/ HTTP/1.1” 200 715 “-” “OpenAPI-G
enerator/3.73.9/ruby”
Oct 6 09:45:06 nj-prod-foreman-proxy01 pulpcore-api[778772]: pulp [0783b25c-7887-44ac-b649-fe7c60188a2c]: - - [06/Oct/2025:09:45:06 +0000] “GET /pulp/api/v3/tasks/0199b8df-28fb-7bb8-a0cb-cbf6cc0a730e/ HTTP/1.1” 200 740 “-” “OpenAPI-G
enerator/3.73.9/ruby”
Oct 6 09:45:07 nj-prod-foreman-proxy01 systemd-coredump[786027]: Process 785448 (pulpcore-worker) of user 982 dumped core.#012#012Stack trace of thread 785448:#012#0 0x00007fe7b5024117 n/a (n/a + 0x0)#012ELF object binary architectur
e: AMD x86-64
Oct 6 09:45:07 nj-prod-foreman-proxy01 systemd[1]: systemd-coredump@7-786026-0.service: Deactivated successfully.
Oct 6 09:45:07 nj-prod-foreman-proxy01 systemd[1]: systemd-coredump@7-786026-0.service: Consumed 3.968s CPU time.
Oct 6 09:45:07 nj-prod-foreman-proxy01 pulpcore-worker-1[1081]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 0199b8de-42f9-7cbf-b18b-5550263e3066 exited with non zero exitcode -11.
Oct 6 09:45:07 nj-prod-foreman-proxy01 pulpcore-worker-1[1081]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 0199b8de-42f9-7cbf-b18b-5550263e3066 in domain: default and marking as failed. Reason: Killed by signal 11.
Oct 6 09:45:07 nj-prod-foreman-proxy01 pulpcore-api[778647]: pulp [0783b25c-7887-44ac-b649-fe7c60188a2c]: - - [06/Oct/2025:09:45:07 +0000] “GET /pulp/api/v3/tasks/0199b8e3-bd26-7573-942f-dea0c8aefd92/ HTTP/1.1” 200 715 “-” “OpenAPI-G
enerator/3.73.9/ruby”
Oct 6 09:45:07 nj-prod-foreman-proxy01 pulpcore-worker-1[786043]: pulp [0783b25c-7887-44ac-b649-fe7c60188a2c]: pulpcore.tasking.tasks:INFO: Starting task 0199b8de-46fe-7ad4-aa52-2ce89c72e049 in domain: default
Oct 6 09:45:07 nj-prod-foreman-proxy01 pulpcore-worker-1[786043]: pulp [0783b25c-7887-44ac-b649-fe7c60188a2c]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=1-Opsview-EL8-splat-prod-f568b1b5-9e34-4e09-a73b-8701472f8
b57 remote=1-Opsview-EL8-splat-prod-f568b1b5-9e34-4e09-a73b-8701472f8b57

4

@ggainey Could this be a pulp bug?

5

Any thoughts on this being a known bug please?

6

Ah, this might be related to Smart proxy sync throws Katello::Errors::Pulp3Error

Are you using the older version of libcomps mentioned in that thread?

7

Reading through that thread it seems very likely
I am running Katello 4.17, and libcomps-0.1.21-3.el9.x86_64

No mention of the affected version of libcomps in that thread other than the fix being in 1.23, so presume mine is affected and needs patching

Are there no other reports of sync failures with foreman release 3.15/Katello 4.17? I upgraded to this a few months ago believing it was a stable fix but looks like I might need to upgrade again soon :frowning:

8

Just realised i still have the current foreman and katello repos on our setup, so will run a dnf update and hopefully pick up this fix. if not, will plan an upgrade next month

9

libcomps-0.1.21-3 is the version that received the seg fault fixes. Foreman and all smart proxies are running that version of libcomps with Pulpcore 3.73? And you’ve restarted all services on each machine to ensure Pulp isn’t using the old libcomps?

The only other reason I’ve seen coredumps like these historically is when the out of memory (OOM) killer comes by. The smart proxy isn’t resource constrained? It might be worth logging the smart proxy’s memory usage during the sync, sometimes the usage can jump up quickly.

If everything above is checked and there are still issues, it would be helpful to run pystack against the pulpcore workers so we can get a proper stack trace. Seeing the core dump might help too.

10

the updated version fixed the sync issue. thanks

1 Like