Problem synchronizing ol7_latest repository on Katello 4.2.1

Support
#1

Problem:

After upgrading Katello 3.18.5 → 4.2.1, I’m unable to successfully synch the ol7_latest repository. I consistently get a checksum error on a package. All other repos are working. I can still synch this repo on a 4.0

A file located at the url http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackageSource/firefox-68.1.0-2.0.1.el7_7.src.rpm failed validation due to checksum.

Same thing if I create a new repo from scratch but the file name is different.

A file located at the url http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackageSource/inkscape-0.91-2.el7.x86_64.rpm failed validation due to checksum.

Foreman and Proxy versions:
foreman-3.0.1-1.el7
katello-4.2.1-1.el7

Distribution and version:
CentOS 7

Other relevant data:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/20749133-ddcf-4e77-a547-8a522340d0f1/",
    "pulp_created"=>"2021-12-16T04:10:02.275+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"788bf2bb-9bde-4c31-a639-275c51577768",
    "started_at"=>"2021-12-16T04:10:02.342+00:00",
    "finished_at"=>"2021-12-16T04:31:42.042+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 317, in _perform_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 490, in synchronize\n" +
       "    version = dv.create()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
       "    loop.run_until_complete(pipeline)\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 174, in run\n" +
       "    pb.done += task.result()  # download_count\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 200, in _handle_content_unit\n" +
       "    await asyncio.gather(*downloaders_for_content)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 89, in download\n" +
       "    download_result = await downloader.run(extra_data=self.extra_data)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 264, in run\n" +
       "    return await download_wrapper()\n" +
       "  File \"/usr/lib/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n" +
       "    ret = await target(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 260, in download_wrapper\n" +
       "    return await self._run(extra_data=extra_data)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py\", line 118, in _run\n" +
       "    to_return = await self._handle_response(response)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 214, in _handle_response\n" +
       "    await self.finalize()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/download/base.py\", line 163, in finalize\n" +
       "    self.validate_digests()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/download/base.py\", line 213, in validate_digests\n" +
       "    raise DigestValidationError(self.url)\n",
      "description"=>
       "A file located at the url https://mirror.netcologne.de/oracle-linux-repos/ol7_latest/getPackage/inkscape-0.91-2.el7.x86_64.rpm failed validation due to checksum."},
    "worker"=>"/pulp/api/v3/workers/042764b4-9f22-4bc5-ae4b-f9738789009b/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>7},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"failed",
       "done"=>4495},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>9501},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "done"=>13014}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/63eea019-10a0-476a-a6de-8806a2a0186d/",
      "/pulp/api/v3/remotes/rpm/rpm/0cad0830-93aa-4e19-aa23-e082ad601995/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>101, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: A file located at the url https://mirror.netcologne.de/oracle-linux-repos/ol7_latest/getPackage/inkscape-0.91-2.el7.x86_64.rpm failed validation due to checksum.
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/pulp3/abstract_async_task.rb:139:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:571:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:565:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:286:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/director.rb:94:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
#2

Testing this now on a 4.0.3 instance to see if I get a different result

#3

Hmm. Different failure on 4.0.3

Response payload is not completed

This must either be an issue at this site or an issue on the repo.

#4

Tested on Katello 4.3RC4

Actions::Pulp3::Repository::Sync
Input:
{"repo_id"=>106,
 "smart_proxy_id"=>1,
 "options"=>{},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"36a10204-d493-4141-a539-4683858b249b",
 "current_timezone"=>"America/New_York",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}
Output:
{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/f85902c5-b704-4463-92a0-71e16c1edbb3/",
    "pulp_created"=>"2021-12-17T14:58:32.618+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"36a10204-d493-4141-a539-4683858b249b",
    "started_at"=>"2021-12-17T14:58:32.816+00:00",
    "finished_at"=>"2021-12-17T15:18:57.868+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x37a78 bytes=106 start="  File \"/u" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x37a8c bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x37aa0 bytes=104 start="  File \"/u" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x37ab4 bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0x37ac8 bytes=109 start="  File \"/u" end="in create\n"> +
       #<Sequel::SQL::Blob:0x37adc bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x37af0 bytes=86 start="  File \"/u" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x37b04 bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x37b18 bytes=102 start="  File \"/u" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x37b2c bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x37b40 bytes=94 start="  File \"/u" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x37b54 bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x37b68 bytes=102 start="  File \"/u" end="3, in run\n"> +
       #<Sequel::SQL::Blob:0x37b7c bytes=47 start="    pb.don" end="oad_count\n"> +
       #<Sequel::SQL::Blob:0x37b90 bytes=119 start="  File \"/u" end="tent_unit\n"> +
       #<Sequel::SQL::Blob:0x37ba4 bytes=51 start="    await " end="_content)\n"> +
       #<Sequel::SQL::Blob:0x37bb8 bytes=97 start="  File \"/u" end=" download\n"> +
       #<Sequel::SQL::Blob:0x37bcc bytes=71 start="    downlo" end="tra_data)\n"> +
       #<Sequel::SQL::Blob:0x37be0 bytes=86 start="  File \"/u" end="7, in run\n"> +
       #<Sequel::SQL::Blob:0x37bf4 bytes=36 start="    return" end="wrapper()\n"> +
       #<Sequel::SQL::Blob:0x37c08 bytes=80 start="  File \"/u" end=" in retry\n"> +
       #<Sequel::SQL::Blob:0x37c1c bytes=40 start="    ret = " end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x37c30 bytes=99 start="  File \"/u" end="d_wrapper\n"> +
       #<Sequel::SQL::Blob:0x37c44 bytes=50 start="    return" end="tra_data)\n"> +
       #<Sequel::SQL::Blob:0x37c58 bytes=89 start="  File \"/u" end=", in _run\n"> +
       #<Sequel::SQL::Blob:0x37c6c bytes=54 start="    to_ret" end="response)\n"> +
       #<Sequel::SQL::Blob:0x37c80 bytes=99 start="  File \"/u" end="_response\n"> +
       #<Sequel::SQL::Blob:0x37c94 bytes=63 start="    chunk " end=" megabyte\n"> +
       #<Sequel::SQL::Blob:0x37ca8 bytes=82 start="  File \"/u" end=", in read\n"> +
       #<Sequel::SQL::Blob:0x37cbc bytes=29 start="    await " end="t(\"read\")\n"> +
       #<Sequel::SQL::Blob:0x37cd0 bytes=83 start="  File \"/u" end=" in _wait\n"> +
       #<Sequel::SQL::Blob:0x37ce4 bytes=17 content="    await waiter\n"> +
       #<Sequel::SQL::Blob:0x37cf8 bytes=86 start="  File \"/u" end=" __exit__\n"> +
       #<Sequel::SQL::Blob:0x37d0c bytes=41 start="    raise " end="from None\n">,
      "description"=>""},
    "worker"=>"/pulp/api/v3/workers/22d4113c-bdbb-4ad8-9de0-6a9b44e778af/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>10},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"failed",
       "done"=>436},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>3500},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "done"=>6507}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/c7d72b28-62ac-44ea-8eab-8831f191f1c8/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/486cbb13-5700-4a3b-ba68-6c65a8459c78/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>96, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: Pulp task error
Backtrace:
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/polling.rb:22:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/cancellable.rb:14:in `run'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.3.0.rc4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/director.rb:94:in `execute'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Action:
Actions::Pulp3::Repository::Sync
Input:
{"repo_id"=>106,
 "smart_proxy_id"=>1,
 "options"=>{},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"36a10204-d493-4141-a539-4683858b249b",
 "current_timezone"=>"America/New_York",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}
Output:
{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/f85902c5-b704-4463-92a0-71e16c1edbb3/",
    "pulp_created"=>"2021-12-17T14:58:32.618+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"36a10204-d493-4141-a539-4683858b249b",
    "started_at"=>"2021-12-17T14:58:32.816+00:00",
    "finished_at"=>"2021-12-17T15:18:57.868+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x37d20 bytes=106 start="  File \"/u" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x37d34 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x37d48 bytes=104 start="  File \"/u" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x37d5c bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0x37d70 bytes=109 start="  File \"/u" end="in create\n"> +
       #<Sequel::SQL::Blob:0x37d84 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x37d98 bytes=86 start="  File \"/u" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x37dac bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x37dc0 bytes=102 start="  File \"/u" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x37dd4 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x37de8 bytes=94 start="  File \"/u" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x37dfc bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x37e10 bytes=102 start="  File \"/u" end="3, in run\n"> +
       #<Sequel::SQL::Blob:0x37e24 bytes=47 start="    pb.don" end="oad_count\n"> +
       #<Sequel::SQL::Blob:0x37e38 bytes=119 start="  File \"/u" end="tent_unit\n"> +
       #<Sequel::SQL::Blob:0x37e4c bytes=51 start="    await " end="_content)\n"> +
       #<Sequel::SQL::Blob:0x37e60 bytes=97 start="  File \"/u" end=" download\n"> +
       #<Sequel::SQL::Blob:0x37e74 bytes=71 start="    downlo" end="tra_data)\n"> +
       #<Sequel::SQL::Blob:0x37e88 bytes=86 start="  File \"/u" end="7, in run\n"> +
       #<Sequel::SQL::Blob:0x37e9c bytes=36 start="    return" end="wrapper()\n"> +
       #<Sequel::SQL::Blob:0x37eb0 bytes=80 start="  File \"/u" end=" in retry\n"> +
       #<Sequel::SQL::Blob:0x37ec4 bytes=40 start="    ret = " end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x37ed8 bytes=99 start="  File \"/u" end="d_wrapper\n"> +
       #<Sequel::SQL::Blob:0x37eec bytes=50 start="    return" end="tra_data)\n"> +
       #<Sequel::SQL::Blob:0x37f00 bytes=89 start="  File \"/u" end=", in _run\n"> +
       #<Sequel::SQL::Blob:0x37f14 bytes=54 start="    to_ret" end="response)\n"> +
       #<Sequel::SQL::Blob:0x37f28 bytes=99 start="  File \"/u" end="_response\n"> +
       #<Sequel::SQL::Blob:0x37f3c bytes=63 start="    chunk " end=" megabyte\n"> +
       #<Sequel::SQL::Blob:0x37f50 bytes=82 start="  File \"/u" end=", in read\n"> +
       #<Sequel::SQL::Blob:0x37f64 bytes=29 start="    await " end="t(\"read\")\n"> +
       #<Sequel::SQL::Blob:0x37f78 bytes=83 start="  File \"/u" end=" in _wait\n"> +
       #<Sequel::SQL::Blob:0x37f8c bytes=17 content="    await waiter\n"> +
       #<Sequel::SQL::Blob:0x37fa0 bytes=86 start="  File \"/u" end=" __exit__\n"> +
       #<Sequel::SQL::Blob:0x37fb4 bytes=41 start="    raise " end="from None\n">,
      "description"=>""},
    "worker"=>"/pulp/api/v3/workers/22d4113c-bdbb-4ad8-9de0-6a9b44e778af/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>10},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"failed",
       "done"=>436},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>3500},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "done"=>6507}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/c7d72b28-62ac-44ea-8eab-8831f191f1c8/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/486cbb13-5700-4a3b-ba68-6c65a8459c78/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>96, "failed"=>1}}
#5

Finally figured this out - it was actually timing out on larger package downloads. I increased sync connection timeout to fix this.

#6

Wow. Glad you worked it out!