Log location for proxy sync error

Support
1

All, I am trying to locate some errors or even warnings when running a full sync for one of our remote smart proxies. I have looked in both “production” and “foreman-proxy” directories on both the master and smart proxy itself, but can’t find anything!
The error comes in at the very end of full sync and stays in the portal until I run an optimized sync - which then works - and all is happy again…

Foreman 3.15, Katello 4.17

Thanks

3

Ok there is an entry in /var/log/messages of all places that shows me the task failure

Sep 30 14:53:38 eq-prod-foreman-proxy01 kernel: traps: pulpcore-worker[1388192] general protection fault ip:7f8f03824117 sp:7ffdd266f7d0 error:0 in libpython3
.12.so.1.0[7f8f036fe000+276000]
Sep 30 14:53:41 eq-prod-foreman-proxy01 pulpcore-worker-2[1091]: pulp [None]: pulpcore.tasking.worker:WARNING: Task process for 01999b14-45ba-741c-b740-0b2839
4858b3 exited with non zero exitcode -11.

I was then able to find the task in Foreman GUI and can now see the detail, albeit I have no idea what’s causing this. All full sync attempts fail, optimised works.

Interested that Foreman and the log entry time stamp has exit code 11 at 14:53 - but that’s only 10 mins after the job started and as below, it officially failed at 16:22. Anyway, if someone could interpret some log details, I would really appreciate it

795: Actions::Pulp3::CapsuleContent::Sync (skipped) [ 5952.19s / 29.03s ]

Queue: default

Started at: 2025-09-30 14:43:39 UTC

Ended at: 2025-09-30 16:22:51 UTC

Real time: 5952.19s

Error:

Katello::Errors::Pulp3Error

Pulp task error

  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:107:in
    `block in check_for_errors’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:105:in
    `each’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:105:in
    `check_for_errors’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:161:in
    `poll_external_task’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/polling.rb:22:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/cancellable.rb:15:in `run’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:590:in `block (3 levels)
    in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:26:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:33:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:16:in
    `block in run’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:40:in
    `block in as_remote_user’”
  • “/usr/share/gems/gems/katello-4.17.0/app/models/katello/concerns/user_extensions.rb:21:in
    `cp_config’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:27:in
    `as_cp_user’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:39:in
    `as_remote_user’”
  • “/usr/share/gems/gems/katello-4.17.0/app/lib/actions/middleware/remote_action.rb:16:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
    `block in run’”
  • “/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in
    `wrap’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/progress.rb:29:in `with_progress_calculation’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action/progress.rb:15:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/load_setting_values.rb:20:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in
    `restore_current_request_id’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in
    `restore_curent_timezone’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
    `restore_current_taxonomies’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:33:in `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:28:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware.rb:20:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in
    `restore_curent_user’”
  • “/usr/share/gems/gems/foreman-tasks-11.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/stack.rb:24:in `call’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/middleware/world.rb:31:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:589:in `block (2 levels)
    in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:588:in `catch’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:588:in `block in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:491:in `block in with_error_handling’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:491:in `catch’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:491:in `with_error_handling’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:583:in `execute_run’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/action.rb:304:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
    `block (2 levels) in execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract.rb:168:in
    `with_meta_calculation’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
    `block in execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
    `open_action’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
    `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/director.rb:95:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in
    `block (2 levels) in perform’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors.rb:18:in `run_user_code’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in
    `block in perform’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in
    `with_telemetry’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
    `perform’”
  • “/usr/share/gems/gems/dynflow-1.9.1/lib/dynflow/executors/sidekiq/serialization.rb:28:in
    `perform’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels)
    in process’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in module:Sidekiq’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog’”
  • “/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread’”
  • “/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block
    in create_with_logging_context’”
4

Is this the same error as from Help with proxy sync failure please - #4 by lfu ?