John,
I’ve been trying to sync these (I’ve formatted them to not be links):
http-mirror-centos-org/centos/6/updates/x86_64/
http-mirror-centos-org/centos/7/os/x86_64/
http-mirror-centos-org/centos/7/opstools/x86_64/
http-mirror-centos-org/centos/7/updates/x86_64/
As a means of troubleshooting, I downloaded centos/6/updates/x86_64/ to an internal webserver then pointed foreman to it. The sync failed with this setup as well. The sync process ran for ~12 minutes before it died. As before, I observed network traffic and storage increase.
I also noticed upwards of 18 connections from the foreman server to the webserver.
…
Jul 17 13:41:08 foreman-01 dynflow-sidekiq@orchestrator: 2020-07-17T18:41:08.313Z 1030 TID-2ya2lq Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-ca8bac8dd19753502afffa35 INFO: start
Jul 17 13:41:08 foreman-01 dynflow-sidekiq@orchestrator: 2020-07-17T18:41:08.315Z 1030 TID-2ya2lq Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-ca8bac8dd19753502afffa35 INFO: done: 0.001 sec
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: pulp: rq.worker:ERROR: aiohttp.client_exceptions.ClientOSError: [Errno 32] Broken pipe
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: Traceback (most recent call last):
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 883, in perform_job
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: rv = job.perform()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 645, in perform
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: self._result = self._execute()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 651, in _execute
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 167, in synchronize
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: dv.create()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: loop.run_until_complete(pipeline)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: return future.result()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await asyncio.gather(*futures)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await self.run()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 133, in run
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: pb.done += task.result() # download_count
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 159, in _handle_content_unit
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await asyncio.gather(*downloaders_for_content)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: download_result = await downloader.run(extra_data=self.extra_data)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: return await self._run(extra_data=extra_data)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/backoff/_async.py”, line 133, in retry
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: ret = await target(*args, **kwargs)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/download/http.py”, line 197, in _run
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1012, in aenter
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: self._resp = await self._coro
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 504, in _request
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await resp.start(conn)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py”, line 847, in start
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: message, payload = await self._protocol.read() # type: ignore # noqa
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 591, in read
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await self._waiter
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: aiohttp.client_exceptions.ClientOSError: [Errno 32] Broken pipe
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: Traceback (most recent call last):
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 883, in perform_job
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: rv = job.perform()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 645, in perform
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: self._result = self._execute()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 651, in _execute
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 167, in synchronize
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: dv.create()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: loop.run_until_complete(pipeline)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: return future.result()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await asyncio.gather(*futures)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await self.run()
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 133, in run
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: pb.done += task.result() # download_count
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 159, in _handle_content_unit
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await asyncio.gather(*downloaders_for_content)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: download_result = await downloader.run(extra_data=self.extra_data)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: return await self._run(extra_data=extra_data)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/backoff/_async.py”, line 133, in retry
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: ret = await target(*args, **kwargs)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/download/http.py”, line 197, in _run
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1012, in aenter
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: self._resp = await self._coro
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 504, in _request
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await resp.start(conn)
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py”, line 847, in start
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: message, payload = await self._protocol.read() # type: ignore # noqa
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 591, in read
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: await self._waiter
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: aiohttp.client_exceptions.ClientOSError: [Errno 32] Broken pipe
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: pulp: rq.worker:INFO: Cleaning registries for queue: 1004@foreman-01.nedisys.local
Jul 17 13:41:20 foreman-01 pulpcore-worker-2: pulp: rq.worker:INFO: 1004@foreman-01.nedisys.local: a0b027f7-38ed-462a-b82c-41d64610619a
Jul 17 13:41:21 foreman-01 pulpcore-worker-2: pulp: rq.worker:INFO: 1004@foreman-01.nedisys.local: Job OK (a0b027f7-38ed-462a-b82c-41d64610619a)
Jul 17 13:41:24 foreman-01 dynflow-sidekiq@worker: 2020-07-17T18:41:24.304Z 1010 TID-2y16sa Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-be22f61c263897cb8e1fb7e6 INFO: start
Jul 17 13:41:24 foreman-01 pulpcore-api: 127.0.0.1 - - [17/Jul/2020:18:41:24 +0000] “GET /pulp/api/v3/tasks/4fc9f8a3-6043-4545-90df-15e2e9aa82de/ HTTP/1.1” 200 3747 “-” “OpenAPI-Generator/3.4.1/ruby”
…
Thanks for your help.