Katello stops syncing with error 'raise asyncio.TimeoutError from None'

Problem:
I configured a new Katello installation, and added all the repositories I need for CentOS 7.
I’m trying to sync 5 new repositories (all of them have been added in the same product CentOS 7).

  • CentOS 7 Base
  • CentOS 7 Extras
  • CentOS 7 Updates
  • EPEL CentOS 7
  • Elastic 7.x

The only repo syncing successfully is CentOS 7 Updates, but I suppose it’s working because I ran it once separately.
All the others repos starts downloading the files from scratch and then stops. All with the same error at the same step: 6: Actions::Pulp3::Repository::Sync (skipped) [ 415.04s / 16.17s ]
I tried to add more RAM to Katello VM (before 8GB, now 10GB) but the result is the same.
Expected outcome:
I expect that the repo sync and download the updates.
Foreman and Proxy versions:
Foreman 2.5.3
Foreman and Proxy plugin versions:
foreman-tasks 4.1.5
foreman_remote_execution 4.5.6
katello 4.1.3
Distribution and version:
CentOS 7.9
Other relevant data:

Output from Dynflow

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/a098a2bb-97e7-49d6-ba2f-8dddd7bf4c42/"
  pulp_created: '2021-09-10T08:30:32.542+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: 45058c47-4ee2-45aa-93dd-7dc118cd1af5
  started_at: '2021-09-10T08:30:32.597+00:00'
  finished_at: '2021-09-10T08:37:24.887+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 489, in synchronize
          version = dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 174, in run
          pb.done += task.result()  # download_count
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 200, in _handle_content_unit
          await asyncio.gather(*downloaders_for_content)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 89, in download
          download_result = await downloader.run(extra_data=self.extra_data)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run
          return await download_wrapper()
        File "/usr/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
          ret = await target(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
          return await self._run(extra_data=extra_data)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 98, in _run
          to_return = await self._handle_response(response)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 210, in _handle_response
          chunk = await response.content.read(1048576)  # 1 megabyte
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 380, in read
          await self._wait("read")
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
          await waiter
        File "/usr/lib64/python3.6/site-packages/aiohttp/helpers.py", line 656, in __exit__
          raise asyncio.TimeoutError from None
    description: ''
  worker: "/pulp/api/v3/workers/ebd061c1-cf7d-4b30-811c-a9d97630ee9f/"
  child_tasks: []
  progress_reports:
  - message: Downloading Metadata Files
    code: sync.downloading.metadata
    state: completed
    done: 6
  - message: Parsed Packages
    code: sync.parsing.packages
    state: completed
    done: 972
  - message: Downloading Artifacts
    code: sync.downloading.artifacts
    state: failed
    done: 23
  - message: Associating Content
    code: associating.content
    state: canceled
    done: 0
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/e3f9c39a-2054-46f3-b443-5be57fce1aac/"
  - "/pulp/api/v3/remotes/rpm/rpm/a1ede720-aa1a-43a3-ace6-b117a9b87e35/"
create_version: true
task_groups: []
poll_attempts:
  total: 45
  failed: 1

@marco.l thanks for bringing this to our attention.

This problem is potentially related to Issue #9301: Users running into timeouts with error messages that are unhelpful - Pulp

As a workaround, please see the comments on Support #33290: Pulp3::Repository::Sync - Timeout - Katello - Foreman and let us know if the timeout increase mentioned there helps you.

@jjeffers I’ve been very busy lately, sorry.
I increased that timeout to 600 and I rerun the sync of only 1 repo (CentOS 7 Base CentOS Mirror).
The sync went a little further, in fact it stopped at
6: Actions::Pulp3::Repository::Sync (skipped) [ 3252.15s / 36.57s ]
which is a lot more than the 415.04 seconds of my last test.

Should I increase again the timeout, maybe 900 or more?

Try that (increasing the timeout) but I am concerned there is something else wrong with access to the repo.