Error while syncing Red Hat repositories

schaudhary · March 30, 2021, 7:34pm

Wondering if anyone has seen the below error while syncing Red Hat repositories

Cannot connect to host cdn.redhat.com:443 ssl:default [None]

I have 5 Red Hat repositories. 2 of those synced successfully. The other 3 keep getting this error. Seems like if the initial sync for a particular repository succeeds, it does not get this error on subsequent syncs.

I have:

Tried removing a repository and adding it again.
Tried setting insecure = 1 in rhsm.conf.
Checked with the network team that the firewall is not blocking the outgoing connections.
Tried different options using Advanced Sync.

jeremylenz · March 31, 2021, 2:00pm

Hi @schaudhary

Is there an error traceback that may provide more information? See if you can check

sudo journalctl -e

when the error happens, and look for that error message.

schaudhary · March 31, 2021, 7:30pm

Jeremy,

Below is what I get from journalctl -e

Mar 31 14:06:35 foremanserver.localpulpcore-api[1037]: - - [31/Mar/2021:19:06:35 +0000] “GET /pulp/api/v3/tasks/aeece1cf-d005-48f6-93c4-854c9862442c/ HTTP/1.1” 200 1441 “-” "OpenAPI-Generator/3.7.1
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 971, in _wrap_create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 820, in create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: sock, protocol_factory, ssl, server_hostname)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 846, in _create_connection_transport
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: yield from waiter
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: ConnectionResetError
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: The above exception was the direct cause of the following exception:
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: Traceback (most recent call last):
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: rv = job.perform()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: self._result = self._execute()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return self.func(*self.args, **self.kwargs)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 271, in synchronize
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: dv.create()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: loop.run_until_complete(pipeline)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return future.result()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: await asyncio.gather(*futures)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: await self.run()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: pb.done += task.result() # download_count
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: await asyncio.gather(*downloaders_for_content)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: download_result = await downloader.run(extra_data=self.extra_data)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return await self._run(extra_data=extra_data)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 88, in _run
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1124, in aenter
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: self._resp = await self._coro
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 528, in _request
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: req, traces=traces, timeout=real_timeout
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 537, in connect
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: proto = await self._create_connection(req, traces, timeout)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 894, in _create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: _, proto = await self._create_direct_connection(req, traces, timeout)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1053, in _create_direct_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: raise last_exc
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1034, in _create_direct_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: client_error=client_error,
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 977, in _wrap_create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: raise client_error(req.connection_key, exc) from exc
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cdn.redhat.com:443 ssl:default [None]
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: Traceback (most recent call last):
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 971, in _wrap_create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 820, in create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: sock, protocol_factory, ssl, server_hostname)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: sock, protocol_factory, ssl, server_hostname)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 846, in _create_connection_transport
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: yield from waiter
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: ConnectionResetError
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: The above exception was the direct cause of the following exception:
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: Traceback (most recent call last):
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: rv = job.perform()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: self._result = self._execute()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return self.func(*self.args, **self.kwargs)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 271, in synchronize
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: dv.create()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: loop.run_until_complete(pipeline)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return future.result()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: await asyncio.gather(*futures)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: await self.run()
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: pb.done += task.result() # download_count
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: await asyncio.gather(*downloaders_for_content)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: download_result = await downloader.run(extra_data=self.extra_data)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: return await self._run(extra_data=extra_data)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 88, in _run
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1124, in aenter
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: self._resp = await self._coro
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 528, in _request
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: req, traces=traces, timeout=real_timeout
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 537, in connect
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: proto = await self._create_connection(req, traces, timeout)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 894, in _create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: _, proto = await self._create_direct_connection(req, traces, timeout)
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1053, in _create_direct_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: raise last_exc
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1034, in _create_direct_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: client_error=client_error,
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 977, in _wrap_create_connection
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: raise client_error(req.connection_key, exc) from exc
Mar 31 14:06:50 foremanserver.localpulpcore-worker-4[1069]: aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cdn.redhat.com:443 ssl:default [None]
Mar 31 14:06:51 foremanserver.localpulpcore-worker-4[1069]: pulp: rq.worker:INFO: Cleaning registries for queue: 1069@av60tpavclfrmn1.flychi.local
Mar 31 14:06:51 foremanserver.localpulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@av60tpavclfrmn1.flychi.local: 1ebcb1d9-28a1-401d-a29a-cdcfa5f86e38
Mar 31 14:06:51 foremanserver.localpulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@av60tpavclfrmn1.flychi.local: Job OK (1ebcb1d9-28a1-401d-a29a-cdcfa5f86e38)
Mar 31 14:06:51 foremanserver.localpulpcore-api[1037]: - - [31/Mar/2021:19:06:51 +0000] “GET /pulp/api/v3/tasks/aeece1cf-d005-48f6-93c4-854c9862442c/ HTTP/1.1” 200 4393 “-” "OpenAPI-Generator/3.7.1
Mar 31 14:06:52 foremanserver.localpulpcore-api[1037]: - - [31/Mar/2021:19:06:52 +0000] “PATCH /pulp/api/v3/distributions/rpm/rpm/614b08b8-9d9f-4a36-8647-6b358b6fd55d/ HTTP/1.1” 202 67 “-” "OpenAPI
Mar 31 14:06:52 foremanserver.localpulpcore-resource-manager[1063]: pulp: rq.worker:INFO: resource-manager: 5b275bc4-a66a-4fd8-ad09-58e621761a3f
Mar 31 14:06:52 foremanserver.localpulpcore-api[1037]: - - [31/Mar/2021:19:06:52 +0000] “GET /pulp/api/v3/tasks/81c8074c-7f08-4988-9178-74949abce12e/ HTTP/1.1” 200 401 “-” "OpenAPI-Generator/3.7.1/
Mar 31 14:06:52 foremanserver.localpulpcore-resource-manager[1063]: pulp: rq.worker:INFO: resource-manager: Job OK (5b275bc4-a66a-4fd8-ad09-58e621761a3f)
Mar 31 14:06:52 foremanserver.localpulpcore-worker-3[1065]: pulp: rq.worker:INFO: 1065@av60tpavclfrmn1.flychi.local: 81c8074c-7f08-4988-9178-74949abce12e
Mar 31 14:06:53 foremanserver.localpulpcore-api[1037]: - - [31/Mar/2021:19:06:53 +0000] “GET /pulp/api/v3/tasks/81c8074c-7f08-4988-9178-74949abce12e/ HTTP/1.1” 200 482 “-” "OpenAPI-Generator/3.7.1/
Mar 31 14:06:53 foremanserver.localpulpcore-api[1037]: - - [31/Mar/2021:19:06:53 +0000] “GET /pulp/api/v3/tasks/81c8074c-7f08-4988-9178-74949abce12e/ HTTP/1.1” 200 482 “-” "OpenAPI-Generator/3.7.1/
Mar 31 14:06:53 foremanserver.localpulpcore-worker-3[1065]: pulp: rq.worker:INFO: 1065@av60tpavclfrmn1.flychi.local: Job OK (81c8074c-7f08-4988-9178-74949abce12e)
Mar 31 14:06:53 foremanserver.localpulpcore-api[1037]: - - [31/Mar/2021:19:06:53 +0000] “GET /pulp/api/v3/tasks/81c8074c-7f08-4988-9178-74949abce12e/ HTTP/1.1” 200 509 “-” "OpenAPI-Generator/3.7.1/
Mar 31 14:06:53 foremanserver.localpulpcore-worker-3[1065]: pulp: rq.worker:INFO: 1065@av60tpavclfrmn1.flychi.local: aac9f8a4-086d-44dc-966d-ab7e10b6b55f
Mar 31 14:06:53 foremanserver.localpulpcore-worker-3[1065]: pulp: rq.worker:INFO: 1065@av60tpavclfrmn1.flychi.local: Job OK (aac9f8a4-086d-44dc-966d-ab7e10b6b55f)

jeremylenz · March 31, 2021, 8:27pm

Are you using an HTTP proxy?

schaudhary · March 31, 2021, 8:35pm

No, no HTTP proxy.

jeremylenz · March 31, 2021, 8:44pm

What do you get when you

curl https://cdn.redhat.com/ -k

from the Foreman server?

schaudhary · March 31, 2021, 9:07pm

curl https://cdn.redhat.com/ -k

Access Denied

You don’t have permission to access “http://cdn.redhat.com/” on this server.

Reference #18.b55a1ab8.1617224774.a91b309

jeremylenz · March 31, 2021, 9:10pm

Ok great, this actually confirms that the server can talk to cdn.redhat.com.

See if a manifest refresh helps: browse to Content > Subscriptions > Manage Manifest > Refresh.

After the refresh, are you able to sync?

schaudhary · March 31, 2021, 9:10pm

Also, as a test, I changed the setting for the RHEL-7-server repo (this is one of the problematic repos)

Verify SSL : No

The sync completed successfully.

So, seems like a ssl configuration issue. Any idea what I could be missing?

schaudhary · March 31, 2021, 9:15pm

I can test it out. However, the server is subscribed directly with Red Hat and there are no content hosts at this time. Does the manifest makes a difference in this situation?

jeremylenz · March 31, 2021, 9:22pm

I think the manifest refresh also regenerates some certificates, so it may help depending on the situation.

Here are some more thoughts:

Can you possibly download the certificate from here:

github.com

Katello/katello/blob/master/ca/redhat-uep.pem

This file has been truncated. show original

and see if you can use it to curl cdn.redhat.com with the --cacert option:

curl https://cdn.redhat.com --cacert <path_to_file>

let’s see if Pulp’s database can tell us something:

sudo -u postgres psql pulpcore -c 'select name, length(ca_cert) from core_remote;'

schaudhary · March 31, 2021, 9:36pm

Ok. Will run these tests tonight and post the results.

schaudhary · April 1, 2021, 12:53am

Jeremy,

Below are 2 repos that were erroring out before and some results.

rhel-7-server-rpms

Normal sync : Error - Cannot connect to host cdn.redhat.com:443 ssl:default [None]

Set Verify SSL = No : Success

Set Verify SSL = Yes : Success

Notes : Seems like once a sync completes successfully, any subsequent syncs do not encounter the error. In this case, once the sync completed with Verify SSL set to No, the subsequent sync succeeded even with Verify SSL set back to Yes.

rhel-7-server-debug-rpms

Normal sync : Error - Server disconnected (different error from the cannot connect to cdn.redhat.com…)

After manifest refresh : Error - Server disconnected (different error from the cannot connect to cdn.redhat.com…)

Curl with cacert :

curl https://cdn.redhat.com --cacert /root/redhat-uep.pem

Access Denied

You don’t have permission to access “http://cdn.redhat.com/” on this server.

Reference #18.b51fc917.1617238214.1bd6db88

Pulp’s database :

sudo -u postgres psql pulpcore -c ‘select name, length(ca_cert) from core_remote;’
name | length
--------------------------------------------------------------------±-------
Red_Hat_Enterprise_Linux_7_Server_Debug_RPMs_x86_64_7Server-287267 | 7731
Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server-274975 | 7731
(2 rows)

Notes : With this repo, I’m getting a different error (server disconnected).

jeremylenz · April 1, 2021, 3:18pm

Ok, the DB query looks fine. And the access denied page is as expected.

Can you give more details about the server disconnected error?

sudo journalctl -u pulpcore-worker@*

schaudhary · April 1, 2021, 4:29pm

Jeremy,

Below are the messages from the sync after the manifest refresh (start around 19:40).

Also, when the scheduled sync ran last night at 21:00, it reverted back to the “Cannot connect to cdn.redhat.com…” error. Messages for that sync are pasted below too.

Mar 31 19:40:19 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: 6f9a26d7-5947-43da-b59b-146ce7271549
Mar 31 19:40:19 foremanserver.local pulpcore-worker-4[1069]: pulp: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=Red_Hat_Enterprise_Linux_7_Server_Debug_RPMs_x86_64_7Server-288066
Mar 31 19:40:56 foremanserver.local pulpcore-worker-1[1072]: pulp: rq.worker:INFO: Cleaning registries for queue: 1072@foremanserver.local
Mar 31 19:45:57 foremanserver.local pulpcore-worker-3[1065]: pulp: rq.worker:INFO: Cleaning registries for queue: 1065@foremanserver.local
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: rv = job.perform()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: self._result = self._execute()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: return self.func(*self.args, **self.kwargs)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 271, in synchronize
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: dv.create()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: loop.run_until_complete(pipeline)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: return future.result()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await asyncio.gather(*futures)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await self.run()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: pb.done += task.result() # download_count
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await asyncio.gather(*downloaders_for_content)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: download_result = await downloader.run(extra_data=self.extra_data)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: return await self._run(extra_data=extra_data)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 88, in _run
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1124, in aenter
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: self._resp = await self._coro
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 551, in _request
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await resp.start(conn)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py”, line 890, in start
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: message, payload = await self._protocol.read() # type: ignore # noqa
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 605, in read
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await self._waiter
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: Traceback (most recent call last):
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: rv = job.perform()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: self._result = self._execute()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: return self.func(*self.args, **self.kwargs)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 271, in synchronize
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: dv.create()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: loop.run_until_complete(pipeline)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: return future.result()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await asyncio.gather(*futures)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await self.run()
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: pb.done += task.result() # download_count
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await asyncio.gather(*downloaders_for_content)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: download_result = await downloader.run(extra_data=self.extra_data)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: return await self._run(extra_data=extra_data)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 88, in _run
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1124, in aenter
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: self._resp = await self._coro
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 551, in _request
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await resp.start(conn)
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py”, line 890, in start
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: message, payload = await self._protocol.read() # type: ignore # noqa
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 605, in read
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: await self._waiter
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: fc5de184-007e-4e5e-8475-70ac5b18063f
Mar 31 19:46:45 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: Job OK (fc5de184-007e-4e5e-8475-70ac5b18063f)
Mar 31 19:47:01 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: 5b387a8c-d491-401e-b300-f93168cbea13
Mar 31 19:47:02 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: Job OK (5b387a8c-d491-401e-b300-f93168cbea13)
Mar 31 19:47:02 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: 5a6eed20-62e1-4d00-b006-46ddba102ebe
Mar 31 19:47:02 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: Job OK (5a6eed20-62e1-4d00-b006-46ddba102ebe)
Mar 31 19:49:43 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: Cleaning registries for queue: 1067@foremanserver.local

Mar 31 21:00:06 foremanserver.local pulpcore-worker-3[1065]: pulp: rq.worker:INFO: 1065@foremanserver.local: 0d4e28ac-a5f4-46a9-880e-23b82d23686e
Mar 31 21:00:06 foremanserver.local pulpcore-worker-3[1065]: pulp: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=Red_Hat_Enterprise_Linux_7_Server_Debug_RPMs_x86_64_7Server-288066
Mar 31 21:00:07 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: 5cd992f0-5af5-4ccb-a2ad-ef1ab10bb4e3
Mar 31 21:00:07 foremanserver.local pulpcore-worker-2[1067]: pulp: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server-279860 remot
Mar 31 21:01:42 foremanserver.local pulpcore-worker-1[1072]: pulp: rq.worker:INFO: Cleaning registries for queue: 1072@foremanserver.local
Mar 31 21:05:53 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: Job OK (5cd992f0-5af5-4ccb-a2ad-ef1ab10bb4e3)
Mar 31 21:05:54 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: Cleaning registries for queue: 1067@foremanserver.local
Mar 31 21:05:54 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: 958faac8-31e6-443e-be7b-d0d0c826d502
Mar 31 21:05:54 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: Job OK (958faac8-31e6-443e-be7b-d0d0c826d502)
Mar 31 21:06:06 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: 2a25830d-8be0-4b40-8a3a-787215d4ed05
Mar 31 21:06:12 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: Job OK (2a25830d-8be0-4b40-8a3a-787215d4ed05)
Mar 31 21:06:12 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: 9081c35e-3be2-4874-b97a-49ddb4b0e3f8
Mar 31 21:06:12 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: 1069@foremanserver.local: Job OK (9081c35e-3be2-4874-b97a-49ddb4b0e3f8)
Mar 31 21:17:48 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: Cleaning registries for queue: 1069@foremanserver.local
Mar 31 21:21:53 foremanserver.local pulpcore-worker-1[1072]: pulp: rq.worker:INFO: Cleaning registries for queue: 1072@foremanserver.local
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 971, in _wrap_create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 820, in create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: sock, protocol_factory, ssl, server_hostname)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 846, in _create_connection_transport
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: yield from waiter
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: ConnectionResetError
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: The above exception was the direct cause of the following exception:
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: Traceback (most recent call last):
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: rv = job.perform()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: self._result = self._execute()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return self.func(*self.args, **self.kwargs)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 271, in synchronize
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: dv.create()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: loop.run_until_complete(pipeline)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return future.result()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: await asyncio.gather(*futures)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: await self.run()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: pb.done += task.result() # download_count
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: await asyncio.gather(*downloaders_for_content)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: download_result = await downloader.run(extra_data=self.extra_data)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return await self._run(extra_data=extra_data)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 88, in _run
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1124, in aenter
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: self._resp = await self._coro
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 528, in _request
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: req, traces=traces, timeout=real_timeout
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 537, in connect
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: proto = await self._create_connection(req, traces, timeout)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: proto = await self._create_connection(req, traces, timeout)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 894, in _create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: _, proto = await self._create_direct_connection(req, traces, timeout)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1053, in _create_direct_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: raise last_exc
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1034, in _create_direct_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: client_error=client_error,
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 977, in _wrap_create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: raise client_error(req.connection_key, exc) from exc
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cdn.redhat.com:443 ssl:default [None]
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: Traceback (most recent call last):
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 971, in _wrap_create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 820, in create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: sock, protocol_factory, ssl, server_hostname)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 846, in _create_connection_transport
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: yield from waiter
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: ConnectionResetError
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: The above exception was the direct cause of the following exception:
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: Traceback (most recent call last):
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: rv = job.perform()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: self._result = self._execute()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return self.func(*self.args, **self.kwargs)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 271, in synchronize
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: dv.create()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: loop.run_until_complete(pipeline)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return future.result()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: await asyncio.gather(*futures)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: await self.run()
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: pb.done += task.result() # download_count
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: await asyncio.gather(*downloaders_for_content)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: download_result = await downloader.run(extra_data=self.extra_data)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: return await self._run(extra_data=extra_data)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 88, in _run
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1124, in aenter
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: self._resp = await self._coro
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 528, in _request
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: req, traces=traces, timeout=real_timeout
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 537, in connect
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: proto = await self._create_connection(req, traces, timeout)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 894, in _create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: _, proto = await self._create_direct_connection(req, traces, timeout)
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1053, in _create_direct_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: raise last_exc
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1034, in _create_direct_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1034, in _create_direct_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: client_error=client_error,
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 977, in _wrap_create_connection
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: raise client_error(req.connection_key, exc) from exc
Mar 31 21:26:06 foremanserver.local pulpcore-worker-3[1065]: aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cdn.redhat.com:443 ssl:default [None]
Mar 31 21:26:07 foremanserver.local pulpcore-worker-3[1065]: pulp: rq.worker:INFO: Cleaning registries for queue: 1065@foremanserver.local
Mar 31 21:26:07 foremanserver.local pulpcore-worker-3[1065]: pulp: rq.worker:INFO: 1065@foremanserver.local: 387ec915-cfa0-4999-af96-e157042b19a7
Mar 31 21:26:07 foremanserver.local pulpcore-worker-3[1065]: pulp: rq.worker:INFO: 1065@foremanserver.local: Job OK (387ec915-cfa0-4999-af96-e157042b19a7)
Mar 31 21:26:10 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: Cleaning registries for queue: 1067@foremanserver.local
Mar 31 21:26:18 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: 00c09792-8b16-497c-8a33-b8f0bce65ce4
Mar 31 21:26:21 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: Job OK (00c09792-8b16-497c-8a33-b8f0bce65ce4)
Mar 31 21:26:21 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: b064453b-9756-4614-ba1d-cd3769458910
Mar 31 21:26:21 foremanserver.local pulpcore-worker-2[1067]: pulp: rq.worker:INFO: 1067@foremanserver.local: Job OK (b064453b-9756-4614-ba1d-cd3769458910)
Mar 31 21:38:02 foremanserver.local pulpcore-worker-4[1069]: pulp: rq.worker:INFO: Cleaning registries for queue: 1069@foremanserver.local

jeremylenz · April 1, 2021, 5:45pm

Thanks @schaudhary

This may be because of too many concurrent downloads. The default is 10 but we should try decreasing that to 5.

For each Red Hat repo, can you try running the following Hammer command:

hammer repository update --id=XXX --download-concurrency=5

(To get a list of id’s, you can run hammer repository list --fields=id,name --search='redhat=true')

After setting download concurrency for all Red Hat repos, let us know if it resolves the sync problems.

Justin_Sherrill · April 1, 2021, 6:27pm

we’ve seen this before as well so I went ahead and opened an issue ( Feature #32246: Support global download_concurrency setting and default to 5 - Katello - Foreman ) to hopefully alleviate this by:
a) lowering the download concurrency to 5 by default
b) adding a global setting so that you do not need to update it per-repo

schaudhary · April 1, 2021, 10:30pm

Tested the actin plan. Received the “Server Disconnected” message during sync. Let me know if you’d like me to provide messages from journalctl, although I expect them to be the same as before.

hammer repository list --fields=id,name --search=‘redhat=true’
—|------------------------------------------------------------

ID	NAME
28	Red Hat Enterprise Linux 7 Server Debug RPMs x86_64 7Server
27	Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server
—	------------------------------------------------------------

hammer repository update --id=28 --download-concurrency=5
Repository updated.

Justin_Sherrill · April 5, 2021, 1:22am

Do you also get the error if you just sync 1 repository?

schaudhary · April 5, 2021, 4:45pm

Justin,

For troubleshooting, I am running the sync one repository at a time only.

Saurabh