Sync EPEL finished with Pulp task error

jrocksk8 · April 13, 2022, 5:29pm

Exception:
Katello::Errors::Pulp3Error: Pulp task error

Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:108:in block in check_for_errors' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in check_for_errors' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:160:in poll_external_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:22:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:94:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in <module:Sidekiq>' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

jeremylenz · April 13, 2022, 5:53pm

Hi @jrocksk8

This is the stack trace from checking the task for errors, but we need to know what the underlying error is.
If you go to the Dynflow console for that task and look for the failed sub-actions, do you see any other errors in the outputs?

jrocksk8 · April 13, 2022, 6:15pm

Hi Jeremy,
This is the Dynflow

[/uploads/default/original/2X/2/2d19e2a14165584ecab1898f5cda41de8064d4f3.png]
This is the “6: Actions” info

6: Actions::Pulp3::Repository::Sync (skipped) [ 337.97s / 3.74s ]
Queue: default

Started at: 2022-04-13 16:55:51 UTC

Ended at: 2022-04-13 17:01:28 UTC

Real time: 337.97s

Execution time (excluding suspended state): 3.74s

Input:

jeremylenz · April 13, 2022, 7:36pm

I still don’t see the error… Can you check /var/log/messages or journalctl -u pulpcore-worker@* ?

jrocksk8 · April 13, 2022, 9:54pm

Okay,
Here you are a new sync:

Apr 13 16:44:31 s273vp01 pulpcore-worker-4[104139]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 074973a8-796f-4b18-84ff-fe5147ffc335
Apr 13 16:44:31 s273vp01 pulpcore-worker-4[104139]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 074973a8-796f-4b18-84ff-fe5147ffc335
Apr 13 16:44:32 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulpcore.tasking.pulpcore_worker:INFO: Starting task b8d00868-a8ab-4a2c-8e6e-4a2fd61a4354
Apr 13 16:44:32 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=EPEL7-21904 remote=EPEL7-21348

Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: task: <Task pending name=‘Task-33’ coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f503b159dc0>()]>>
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: task: <Task pending name=‘Task-36’ coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f503b159e50>()]>>
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: task: <Task pending name=‘Task-38’ coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f503b159eb0>()]>>
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: task: <Task pending name=‘Task-40’ coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f503b159f10>()]>>
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: task: <Task pending name=‘Task-42’ coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f503b159f70>()]>>
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: task: <Task pending name=‘Task-44’ coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f503b159fd0>()]>>
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: task: <Task pending name=‘Task-48’ coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f503b14bd00>()]>>
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulpcore.tasking.pulpcore_worker:INFO: Task b8d00868-a8ab-4a2c-8e6e-4a2fd61a4354 failed ()
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulpcore.tasking.pulpcore_worker:INFO: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py”, line 370, in _perform_task
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: result = func(*args, **kwargs)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 535, in synchronize
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: repo_version = dv.create() or repo.latest_version()
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 161, in create
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: loop.run_until_complete(pipeline)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py”, line 616, in run_until_complete
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: return future.result()
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: await asyncio.gather(*futures)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: await self.run()
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 183, in run
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: pb.done += task.result() # download_count
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 209, in _handle_content_unit
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: await asyncio.gather(*downloaders_for_content)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/models.py”, line 89, in download
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: download_result = await downloader.run(extra_data=self.extra_data)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py”, line 247, in run
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: return await download_wrapper()
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/backoff/_async.py”, line 133, in retry
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: ret = await target(*args, **kwargs)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py”, line 245, in download_wrapper
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: return await self._run(extra_data=extra_data)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py”, line 118, in _run
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: to_return = await self._handle_response(response)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py”, line 199, in _handle_response
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: chunk = await response.content.read(1048576) # 1 megabyte
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/streams.py”, line 380, in read
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: await self._wait(“read”)
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/streams.py”, line 306, in _wait
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: await waiter
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: File “/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/helpers.py”, line 656, in exit
Apr 13 16:50:12 s273vp01 pulpcore-worker-1[104138]: raise asyncio.TimeoutError from None
Apr 13 16:50:25 s273vp01 pulpcore-worker-4[104139]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 46fddcda-1250-40ab-acd1-74903ca4fdb2
Apr 13 16:50:25 s273vp01 pulpcore-worker-4[104139]: pulp [9d2a9e1b-8dd2-4394-90b0-07bfca9b85cb]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 46fddcda-1250-40ab-acd1-74903ca4fdb2

jeremylenz · April 13, 2022, 10:00pm

Looks like that’s the underlying error. @dralley thoughts? Is it related to Issue #9009: CacheBasicOperationsTestCase::test_04_basic_expires unit test fails intermittently - Pulp ?

dralley · April 13, 2022, 10:18pm

No, the underlying error is actually the TimeoutError, which may fall under Users running into timeouts with error messages that are unhelpful · Issue #2169 · pulp/pulpcore · GitHub

What version of pulpcore do you have running?
How fast is your internet connection?
Does increasing the timeout value on the remote fix the problem?

jrocksk8 · April 21, 2022, 7:32pm

Sorry by the delay…

tfm-pulpcore-python3-pulpcore-3.16.1-1.el7.noarch
20MB/s shared (I saw a good interface throughput while sync is runnig)
I’ve just increased the “Sync Connection Timeout” from 300 to 600. I am waiting that my sync finish.

I will keep you informed.

jrocksk8 · April 25, 2022, 8:51pm

Hello guys,
Effectively, increasing the “Sync Connection Timeout” to 600 solved my sync issue.
Thank you.