Katello::Errors::Pulp3Error: Response payload is not completed

Problem:
When I try to sync repo it shows errors like as follow,
Katello::Errors::Pulp3Error: Response payload is not completed

Every time i’m facing this error after installing Foreman 2.4.0. Please some one save me!!

Expected outcome:

Foreman and Proxy versions:

  • foreman-2.4.0-1.el8.noarch
  • foreman-proxy-2.4.0-1.el8.noarch
  • katello-4.0.1-1.el8.noarch

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 8.3.2011

Other relevant data:

Screenshot:
image

Input:

{“repo_id”=>11,
“smart_proxy_id”=>1,
“options”=>{“source_url”=>nil},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>“74bda4f4-8cd6-4560-871e-2736d6053eb7”,
“current_timezone”=>“Asia/Dhaka”,
“current_organization_id”=>1,
“current_location_id”=>2,
“current_user_id”=>4}

output:

{“pulp_tasks”=>
[{“pulp_href”=>“/pulp/api/v3/tasks/f84c30d5-9e91-450f-b3e3-44065c374512/”,
“pulp_created”=>“2021-06-08T13:29:08.621+00:00”,
“state”=>“failed”,
“name”=>“pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”=>“a4de8917bb1344d4b82ebf68c0ddc2e6”,
“started_at”=>“2021-06-08T13:29:09.213+00:00”,
“finished_at”=>“2021-06-08T13:37:06.095+00:00”,
“error”=>
{“traceback”=>
" File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job\n" +
" rv = job.perform()\n" +
" File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform\n" +
" self._result = self._execute()\n" +
" File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize\n" +
" dv.create()\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete\n" +
" return future.result()\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in call\n" +
" await self.run()\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 152, in run\n" +
" pb.done += task.result() # download_count\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 178, in _handle_content_unit\n" +
" await asyncio.gather(*downloaders_for_content)\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 88, in download\n" +
" download_result = await downloader.run(extra_data=self.extra_data)\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/download/base.py", line 227, in run\n" +
" return await self._run(extra_data=extra_data)\n" +
" File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 90, in _run\n" +
" to_return = await self._handle_response(response)\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 180, in _handle_response\n" +
" chunk = await response.content.read(1048576) # 1 megabyte\n" +
" File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 381, in read\n" +
" await self._wait("read")\n" +
" File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 307, in _wait\n" +
" await waiter\n",
“description”=>“Response payload is not completed”},
“worker”=>“/pulp/api/v3/workers/62f54fcd-0aa0-4b94-9bbb-235f3c6577e6/”,
“child_tasks”=>,
“progress_reports”=>
[{“message”=>“Parsed Comps”,
“code”=>“parsing.comps”,
“state”=>“completed”,
“total”=>91,
“done”=>91},
{“message”=>“Parsed Advisories”,
“code”=>“parsing.advisories”,
“state”=>“completed”,
“total”=>4792,
“done”=>4792},
{“message”=>“Downloading Artifacts”,
“code”=>“downloading.artifacts”,
“state”=>“failed”,
“done”=>498},
{“message”=>“Parsed Packages”,
“code”=>“parsing.packages”,
“state”=>“canceled”,
“total”=>31782,
“done”=>12013},
{“message”=>“Downloading Metadata Files”,
“code”=>“downloading.metadata”,
“state”=>“canceled”,
“done”=>5},
{“message”=>“Associating Content”,
“code”=>“associating.content”,
“state”=>“canceled”,
“done”=>14000},
{“message”=>“Un-Associating Content”,
“code”=>“unassociating.content”,
“state”=>“canceled”,
“done”=>0}],
“created_resources”=>,
“reserved_resources_record”=>
[“/pulp/api/v3/repositories/rpm/rpm/cb5d1c86-f09b-4b79-8845-070441340436/”,
“/pulp/api/v3/remotes/rpm/rpm/f643901f-0947-421e-b1e4-5bd58a57b3e7/”]}],
“create_version”=>true,
“task_groups”=>,
“poll_attempts”=>{“total”=>50, “failed”=>1}}

Exception:

Katello::Errors::Pulp3Error: Response payload is not completed

can you try to adjust the download_concurrency to 1:

hammer repository update --download-concurrency=1 --id=REPO_ID

where REPO_ID is the id of the repo which you should be able to find by looking at the url when you are on the repo details page: /products/1/repositories/REPO_ID. Alternatively you can use the --name (alongside --product) to specify the repo.

Then try to sync the repository again.

Thanks, I’ll try this method & let you know the update

Sometimes this type of error is shown

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: Error reading from remote serve

And you are using an http proxy?

As far i know i didn’t use that kind of thing. May i give any log which you can see and got my actual problem? It will be great helpful for me.


Thanks for your kind reply.

i would check ‘journalctl -u pulpcore-worker@*’

and look for any errors/tracebacks there. To figure out if you are using a proxy server, in the UI go to Administer > settings > content tab > look for ‘Default HTTP proxy’

If its blank, there’s no proxy server configured (but then i’m really confused about your error message).

journalctl -u pulpcore-worker@

– Logs begin at Tue 2021-06-08 18:41:57 +06, end at Tue 2021-06-08 21:56:09 +06. –
– No entries –

Didn’t produce loge for me.

proxy

you’re missing the ‘*’ on that journalctl command:

journalctl -u 'pulpcore-worker@*'

Sorry for my mistake.

Logs:

– Logs begin at Tue 2021-06-08 18:41:57 +06, end at Tue 2021-06-08 22:02:05 +06. –
Jun 08 18:42:12 foreman.test01.com systemd[1]: Started Pulp RQ Worker.
Jun 08 18:42:12 foreman.test01.com systemd[1]: Started Pulp RQ Worker.
Jun 08 18:42:12 foreman.test01.com systemd[1]: Started Pulp RQ Worker.
Jun 08 18:42:12 foreman.test01.com systemd[1]: Started Pulp RQ Worker.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker ‘1332@foreman.test01.com’.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker ‘1330@foreman.test01.com’.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker ‘1327@foreman.test01.com’.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker ‘1331@foreman.test01.com’.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Worker rq:worker:1330@foreman.test01.com: started, version 1.7.0
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:1330@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Worker rq:worker:1332@foreman.test01.com: started, version 1.7.0
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:1332@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Worker rq:worker:1331@foreman.test01.com: started, version 1.7.0
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:1331@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: *** Listening on 1330@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: *** Listening on 1332@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: *** Listening on 1331@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Worker rq:worker:1327@foreman.test01.com: started, version 1.7.0
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:1327@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: *** Listening on 1327@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker ‘1332@foreman.test01.com’ discovered
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker ‘1327@foreman.test01.com’ discovered
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker ‘1331@foreman.test01.com’ discovered
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker ‘1330@foreman.test01.com’ discovered
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:42:30 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:44:10 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:44:10 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:44:10 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:44:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘1337@foreman.test01.com’ has gone missing, removing from list of workers
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1337@foreman.test01.com is missing. Canceling the tasks in its queue.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘1337@foreman.test01.com’ has gone missing, removing from list of workers
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1337@foreman.test01.com is missing. Canceling the tasks in its queue.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘1339@foreman.test01.com’ has gone missing, removing from list of workers
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1339@foreman.test01.com is missing. Canceling the tasks in its queue.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘1339@foreman.test01.com’ has gone missing, removing from list of workers
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1339@foreman.test01.com is missing. Canceling the tasks in its queue.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘1335@foreman.test01.com’ has gone missing, removing from list of workers
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1335@foreman.test01.com is missing. Canceling the tasks in its queue.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘1335@foreman.test01.com’ has gone missing, removing from list of workers
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1335@foreman.test01.com is missing. Canceling the tasks in its queue.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:45:51 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jun 08 18:56:54 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: 454aae28-709a-4b30-a061-24530cea005c
Jun 08 18:56:54 foreman.test01.com pulpcore-worker-3[1330]: pulp [a9253da70d7843a791a9ea50be883db6]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (454aae28-709a-4b30-a061-24530cea005c)
Jun 08 18:56:54 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: 90b0f816-d47c-4928-935f-40ec829ce247
Jun 08 18:56:54 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (90b0f816-d47c-4928-935f-40ec829ce247)
Jun 08 18:57:59 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: d8abcb98-6c1b-490c-a609-d003471f2ec2
Jun 08 18:58:00 foreman.test01.com pulpcore-worker-3[1330]: pulp [777c711425ac4cf6a9120b174d27ae94]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (d8abcb98-6c1b-490c-a609-d003471f2ec2)
Jun 08 18:58:00 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: e2b25251-1ffb-4a2d-85a6-86cbc65e8614
Jun 08 18:58:00 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (e2b25251-1ffb-4a2d-85a6-86cbc65e8614)
Jun 08 18:58:01 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: eb7bc30f-0784-4987-a0ac-41c3697c36f9
Jun 08 18:58:01 foreman.test01.com pulpcore-worker-3[1330]: pulp [eb12430ff0f4496494f3b1e225e72a62]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (eb7bc30f-0784-4987-a0ac-41c3697c36f9)
Jun 08 18:58:01 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: 2a839e1a-cc6f-47b1-a848-f3ee5013882c
Jun 08 18:58:01 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: 21ef2a68-86a0-43b6-a6f6-affd52ddf1d0
Jun 08 18:58:01 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (21ef2a68-86a0-43b6-a6f6-affd52ddf1d0)
Jun 08 18:58:02 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 1ee650c8-1441-463d-9001-d70d0f500966
Jun 08 18:58:02 foreman.test01.com pulpcore-worker-4[1327]: pulp [80c13806d59f4ea4b8ebf4a0b39c7d1f]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (1ee650c8-1441-463d-9001-d70d0f500966)
Jun 08 18:58:02 foreman.test01.com pulpcore-worker-1[1331]: pulp [050faac679e946848494ceaad7b90b87]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (2a839e1a-cc6f-47b1-a848-f3ee5013882c)
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: ccabe01d-c145-4450-8587-fe40638dd57f
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: bd14704e-ece6-4d06-814b-25062026dd56
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (bd14704e-ece6-4d06-814b-25062026dd56)
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (ccabe01d-c145-4450-8587-fe40638dd57f)
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: 6dc345cd-58c9-4e30-8345-94b4a38c37f3
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-3[1330]: pulp [142b997eade749fb8595d6522a6bf9e0]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (6dc345cd-58c9-4e30-8345-94b4a38c37f3)
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 7a5b035b-b998-4a37-86cc-fef3756970bd
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: f88d7942-a3ca-4e6e-afe9-c069f5b8f3d5
Jun 08 18:58:03 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (f88d7942-a3ca-4e6e-afe9-c069f5b8f3d5)
Jun 08 18:58:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [4fde32f58f314f8196afaa0931879aef]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (7a5b035b-b998-4a37-86cc-fef3756970bd)
Jun 08 18:58:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: fcb9547d-723b-45f3-8b3a-85db464baf21
Jun 08 18:58:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (fcb9547d-723b-45f3-8b3a-85db464baf21)
Jun 08 18:58:04 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: af76f46d-27d1-43f7-8470-d0fb320c50c5
Jun 08 18:58:05 foreman.test01.com pulpcore-worker-4[1327]: pulp [e4fc78e144cf4913a995e80c0be4b923]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (af76f46d-27d1-43f7-8470-d0fb320c50c5)
Jun 08 18:58:05 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 8694b0a4-6fb7-4f53-9e93-1df6c11ba8ff
Jun 08 18:58:05 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (8694b0a4-6fb7-4f53-9e93-1df6c11ba8ff)
Jun 08 19:02:26 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: c4f02cd0-b0de-473b-bfab-1dff17e14199
Jun 08 19:02:27 foreman.test01.com pulpcore-worker-3[1330]: pulp [be60de17fe6d45ea85b510b29e34da1f]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=RHEL-8-REPO-AppStream-107833, version=0
Jun 08 19:02:28 foreman.test01.com pulpcore-worker-3[1330]: pulp [be60de17fe6d45ea85b510b29e34da1f]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (c4f02cd0-b0de-473b-bfab-1dff17e14199)
Jun 08 19:02:28 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: d75a983f-bf49-4373-b42b-41a541927c40
Jun 08 19:02:28 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (d75a983f-bf49-4373-b42b-41a541927c40)
Jun 08 19:02:29 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 981572f7-db2f-4a0e-a363-177c7a243757
Jun 08 19:02:29 foreman.test01.com pulpcore-worker-2[1332]: pulp [bd45d2916ca04876873c2ffda9e9b52e]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (981572f7-db2f-4a0e-a363-177c7a243757)
Jun 08 19:02:29 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 1b478e5b-0b5c-4972-a974-dbcf58ea3ecf
Jun 08 19:02:30 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (1b478e5b-0b5c-4972-a974-dbcf58ea3ecf)
Jun 08 19:02:35 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 9693e679-231d-4c4f-8452-91d881b4cfaf
Jun 08 19:02:35 foreman.test01.com pulpcore-worker-4[1327]: pulp [04c8ad9a6d1148028d3a2ea6fcb4cb6b]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=RHEL-8-REPO-AppStream-107833 remote=RHEL-8-REPO-AppStream-108718
Jun 08 19:03:04 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 19:04:09 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 19:04:10 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 19:21:09 foreman.test01.com pulpcore-worker-4[1327]: pulp [04c8ad9a6d1148028d3a2ea6fcb4cb6b]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (9693e679-231d-4c4f-8452-91d881b4cfaf)
Jun 08 19:21:09 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 19:21:09 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 73866752-51e5-4c60-b4a8-1e94dead40e0
Jun 08 19:21:09 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (73866752-51e5-4c60-b4a8-1e94dead40e0)
Jun 08 19:21:13 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 4be067dd-2731-4f8e-a4fd-46f2d77ac21d
Jun 08 19:21:13 foreman.test01.com pulpcore-worker-4[1327]: pulp [1f2cf0bb9f5e4a4ea254e3a7d7ca0919]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=RHEL-8-REPO-AppStream-107833, version=1
Jun 08 19:23:05 foreman.test01.com pulpcore-worker-4[1327]: pulp [1f2cf0bb9f5e4a4ea254e3a7d7ca0919]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (4be067dd-2731-4f8e-a4fd-46f2d77ac21d)
Jun 08 19:23:05 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 7317e450-fb78-4f1c-933a-7a72411fd515
Jun 08 19:23:06 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (7317e450-fb78-4f1c-933a-7a72411fd515)
Jun 08 19:23:10 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: e603508f-eea8-4718-a91f-65139051072c
Jun 08 19:23:11 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 19:23:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [6fe625b1d7944ec7997201915a95ef51]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (e603508f-eea8-4718-a91f-65139051072c)
Jun 08 19:23:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: af3619d7-5e63-42b6-8adc-d63d5cc87783
Jun 08 19:23:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (af3619d7-5e63-42b6-8adc-d63d5cc87783)
Jun 08 19:23:12 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: e072db34-9991-4251-ba9e-32d3faf2fea4
Jun 08 19:23:13 foreman.test01.com pulpcore-worker-2[1332]: pulp [73821125913b4120bb5bbd3278481b8d]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (e072db34-9991-4251-ba9e-32d3faf2fea4)
Jun 08 19:23:13 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: e5fafa56-8fe1-49c8-b3bb-9eb33dae1a17
Jun 08 19:23:13 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (e5fafa56-8fe1-49c8-b3bb-9eb33dae1a17)
Jun 08 19:24:16 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 19:24:53 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 19:28:42 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 5ca5e78c-39be-4fc4-a57f-be01e76c1017
Jun 08 19:28:42 foreman.test01.com pulpcore-worker-2[1332]: pulp [36789bc02264401eb3afd075cc086835]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=RHEL-7-REPO-118599, version=0
Jun 08 19:28:43 foreman.test01.com pulpcore-worker-2[1332]: pulp [36789bc02264401eb3afd075cc086835]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (5ca5e78c-39be-4fc4-a57f-be01e76c1017)
Jun 08 19:28:43 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 927ed7c0-41e1-47e0-8ab8-e0f8b85dad65
Jun 08 19:28:43 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (927ed7c0-41e1-47e0-8ab8-e0f8b85dad65)
Jun 08 19:28:44 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: d2c00076-64e3-4e1d-a449-90b9e611660a
Jun 08 19:28:46 foreman.test01.com pulpcore-worker-1[1331]: pulp [00d01a528d0148d5b55995ea73560ccd]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (d2c00076-64e3-4e1d-a449-90b9e611660a)
Jun 08 19:28:46 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: 22d6289e-726d-4105-b70e-13e13236645e
Jun 08 19:28:46 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (22d6289e-726d-4105-b70e-13e13236645e)
Jun 08 19:29:08 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: f84c30d5-9e91-450f-b3e3-44065c374512
Jun 08 19:29:09 foreman.test01.com pulpcore-worker-3[1330]: pulp [a4de8917bb1344d4b82ebf68c0ddc2e6]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=RHEL-7-REPO-118599 remote=RHEL-7-REPO-119401
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: pulp [a4de8917bb1344d4b82ebf68c0ddc2e6]: rq.worker:ERROR: Traceback (most recent call last):
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 975, in perform_job
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: rv = job.perform()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 696, in perform
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: self._result = self._execute()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 719, in _execute
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: return self.func(*self.args, **self.kwargs)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 269, in synchronize
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: dv.create()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: loop.run_until_complete(pipeline)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: return future.result()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await asyncio.gather(*futures)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await self.run()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: pb.done += task.result() # download_count
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await asyncio.gather(*downloaders_for_content)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: download_result = await downloader.run(extra_data=self.extra_data)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: return await self._run(extra_data=extra_data)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 90, in _run
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: to_return = await self._handle_response(response)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/download/http.py”, line 180, in _handle_response
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: chunk = await response.content.read(1048576) # 1 megabyte
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 381, in read
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await self._wait(“read”)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 307, in _wait
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await waiter
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: Traceback (most recent call last):
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 975, in perform_job
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: rv = job.perform()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 696, in perform
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: self._result = self._execute()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 719, in _execute
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: return self.func(*self.args, **self.kwargs)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 269, in synchronize
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: dv.create()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: loop.run_until_complete(pipeline)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: return future.result()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await asyncio.gather(*futures)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await self.run()
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: pb.done += task.result() # download_count
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await asyncio.gather(*downloaders_for_content)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: download_result = await downloader.run(extra_data=self.extra_data)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: return await self._run(extra_data=extra_data)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 90, in _run
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: to_return = await self._handle_response(response)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib/python3.6/site-packages/pulpcore/download/http.py”, line 180, in _handle_response
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: chunk = await response.content.read(1048576) # 1 megabyte
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 381, in read
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await self._wait(“read”)
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 307, in _wait
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: await waiter
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: 14f7956d-d51e-4276-a7f1-006e03cab50a
Jun 08 19:37:06 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (14f7956d-d51e-4276-a7f1-006e03cab50a)
Jun 08 19:37:24 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: a6abd1b6-9c88-4011-a3ce-539d7e72deb4
Jun 08 19:37:26 foreman.test01.com pulpcore-worker-1[1331]: pulp [a1268a55f0394e95a87787aa81c1ebc8]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (a6abd1b6-9c88-4011-a3ce-539d7e72deb4)
Jun 08 19:37:26 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: 6c217d34-78b4-4d70-a8e2-13996add4192
Jun 08 19:37:26 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (6c217d34-78b4-4d70-a8e2-13996add4192)
Jun 08 19:41:35 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 19:44:08 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 19:45:27 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 19:47:08 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 20:01:38 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 20:04:11 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 20:05:05 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: c4550cba-8810-4ac4-ac8e-9c9a8d27df59
Jun 08 20:05:06 foreman.test01.com pulpcore-worker-2[1332]: pulp [12516b05968f4544babbc72a2347305f]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=RHEL-8-REPO-BaseOS-122797, version=0
Jun 08 20:05:07 foreman.test01.com pulpcore-worker-2[1332]: pulp [12516b05968f4544babbc72a2347305f]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (c4550cba-8810-4ac4-ac8e-9c9a8d27df59)
Jun 08 20:05:07 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: a489f3cb-3901-42dd-b40f-eff8e9ff124a
Jun 08 20:05:07 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (a489f3cb-3901-42dd-b40f-eff8e9ff124a)
Jun 08 20:05:16 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: 63ea3eb8-2082-448c-9ab7-10f628ed579f
Jun 08 20:05:16 foreman.test01.com pulpcore-worker-1[1331]: pulp [0fc63b478a3140f38873d61d07833e32]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=RHEL-8-REPO-BaseOS-122797 remote=RHEL-8-REPO-BaseOS-121352
Jun 08 20:06:48 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 20:07:11 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 20:09:01 foreman.test01.com pulpcore-worker-1[1331]: pulp [0fc63b478a3140f38873d61d07833e32]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (63ea3eb8-2082-448c-9ab7-10f628ed579f)
Jun 08 20:09:01 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: 8e8065fe-51a3-4865-9160-df24270c312e
Jun 08 20:09:01 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (8e8065fe-51a3-4865-9160-df24270c312e)
Jun 08 20:09:04 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 8b7a97c9-7206-4670-90cf-f84e573971b5
Jun 08 20:09:05 foreman.test01.com pulpcore-worker-4[1327]: pulp [1d0a29a1a6274f93a148e9fa488c43f0]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=RHEL-8-REPO-BaseOS-122797, version=1
Jun 08 20:12:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [1d0a29a1a6274f93a148e9fa488c43f0]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (8b7a97c9-7206-4670-90cf-f84e573971b5)
Jun 08 20:12:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 809ef983-964f-4585-b67d-93b07e0868a8
Jun 08 20:12:30 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (809ef983-964f-4585-b67d-93b07e0868a8)
Jun 08 20:12:40 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 04fe62ba-435b-4262-a880-0fd0e1661494
Jun 08 20:12:50 foreman.test01.com pulpcore-worker-2[1332]: pulp [fdbbbb4d50d242878a587a86f4eeb956]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (04fe62ba-435b-4262-a880-0fd0e1661494)
Jun 08 20:12:50 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 7eb19538-511e-48ae-9093-8db80965b66f
Jun 08 20:12:52 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (7eb19538-511e-48ae-9093-8db80965b66f)
Jun 08 20:12:52 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 46478711-8dcd-41c1-8231-4e3dda322a09
Jun 08 20:12:53 foreman.test01.com pulpcore-worker-2[1332]: pulp [f07d420f8dd8412096a1fd3858b51e91]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (46478711-8dcd-41c1-8231-4e3dda322a09)
Jun 08 20:12:53 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 66e3c0b3-6c52-4f88-991a-e145f6e8f2e1
Jun 08 20:12:53 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (66e3c0b3-6c52-4f88-991a-e145f6e8f2e1)
Jun 08 20:22:33 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 20:25:47 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 20:27:17 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 20:27:58 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 20:42:40 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 20:45:53 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 20:47:24 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 20:48:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 21:02:46 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 21:06:00 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 21:07:30 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 21:08:11 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 21:19:47 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: fa164e41-8c43-4689-9be1-06d97dc81cd3
Jun 08 21:19:48 foreman.test01.com pulpcore-worker-2[1332]: pulp [a6fb99ac701348ccb1cd3d7271977e12]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (fa164e41-8c43-4689-9be1-06d97dc81cd3)
Jun 08 21:19:48 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: b49ba58e-ad45-4fbf-a322-c1fa3a0f795f
Jun 08 21:19:48 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (b49ba58e-ad45-4fbf-a322-c1fa3a0f795f)
Jun 08 21:19:49 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: c83f56a5-2494-4e21-ba64-98f6ed3c7e45
Jun 08 21:19:49 foreman.test01.com pulpcore-worker-3[1330]: pulp [5b7c6599c3034ed1a7fe9b9d38aee931]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (c83f56a5-2494-4e21-ba64-98f6ed3c7e45)
Jun 08 21:19:49 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 96ee651f-2a9a-4e80-befe-eb436de8b994
Jun 08 21:19:49 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: f56b960d-d9ec-4342-b004-3ab922059a3a
Jun 08 21:19:49 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (f56b960d-d9ec-4342-b004-3ab922059a3a)
Jun 08 21:19:49 foreman.test01.com pulpcore-worker-2[1332]: pulp [ad09c047ef174d4cb1d50b855def1772]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (96ee651f-2a9a-4e80-befe-eb436de8b994)
Jun 08 21:19:49 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 66e34861-0ed4-4b06-99e0-d4858b8becc7
Jun 08 21:19:50 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (66e34861-0ed4-4b06-99e0-d4858b8becc7)
Jun 08 21:22:37 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 282460ab-69b8-4bcb-b2ff-3fb15589919e
Jun 08 21:22:37 foreman.test01.com pulpcore-worker-2[1332]: pulp [f40f4df8bac743bda6259ae31b58a575]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=RHEL-7-REPO-131233, version=0
Jun 08 21:22:38 foreman.test01.com pulpcore-worker-2[1332]: pulp [f40f4df8bac743bda6259ae31b58a575]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (282460ab-69b8-4bcb-b2ff-3fb15589919e)
Jun 08 21:22:38 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 621e8a22-b568-4e4f-8426-c8238ba341ce
Jun 08 21:22:38 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (621e8a22-b568-4e4f-8426-c8238ba341ce)
Jun 08 21:22:39 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 68723d8a-bbf8-4f08-aeae-46f221945843
Jun 08 21:22:39 foreman.test01.com pulpcore-worker-4[1327]: pulp [367b98ab2dfd498092e62f0a68c3b5a9]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (68723d8a-bbf8-4f08-aeae-46f221945843)
Jun 08 21:22:39 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: b05deb48-8a51-4e88-a629-f5cc02544d2a
Jun 08 21:22:40 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (b05deb48-8a51-4e88-a629-f5cc02544d2a)
Jun 08 21:22:47 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 37b2495d-9965-403f-b747-8e00f42f4078
Jun 08 21:22:47 foreman.test01.com pulpcore-worker-2[1332]: pulp [0e33dc94ca874e03869adff635422806]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=RHEL-7-REPO-131233 remote=RHEL-7-REPO-137732
Jun 08 21:24:20 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 21:26:05 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 21:28:11 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [0e33dc94ca874e03869adff635422806]: rq.worker:ERROR: Traceback (most recent call last):
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 975, in perform_job
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: rv = job.perform()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 696, in perform
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: self._result = self._execute()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 719, in _execute
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: return self.func(*self.args, **self.kwargs)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 269, in synchronize
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: dv.create()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: loop.run_until_complete(pipeline)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: return future.result()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await asyncio.gather(*futures)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await self.run()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: pb.done += task.result() # download_count
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await asyncio.gather(*downloaders_for_content)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: download_result = await downloader.run(extra_data=self.extra_data)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: return await self._run(extra_data=extra_data)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 90, in _run
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: to_return = await self._handle_response(response)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/download/http.py”, line 180, in _handle_response
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: chunk = await response.content.read(1048576) # 1 megabyte
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 381, in read
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await self._wait(“read”)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 307, in _wait
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await waiter
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: Traceback (most recent call last):
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 975, in perform_job
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: rv = job.perform()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 696, in perform
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: self._result = self._execute()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 719, in _execute
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: return self.func(*self.args, **self.kwargs)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 269, in synchronize
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: dv.create()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: loop.run_until_complete(pipeline)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: return future.result()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await asyncio.gather(*futures)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await self.run()
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: pb.done += task.result() # download_count
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await asyncio.gather(*downloaders_for_content)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: download_result = await downloader.run(extra_data=self.extra_data)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: return await self._run(extra_data=extra_data)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 90, in _run
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: to_return = await self._handle_response(response)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib/python3.6/site-packages/pulpcore/download/http.py”, line 180, in _handle_response
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: chunk = await response.content.read(1048576) # 1 megabyte
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 381, in read
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await self._wait(“read”)
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: File “/usr/lib64/python3.6/site-packages/aiohttp/streams.py”, line 307, in _wait
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: await waiter
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1332@foreman.test01.com
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: b8746e66-f8f8-4204-ba6e-ffaeae7d3649
Jun 08 21:38:04 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: Job OK (b8746e66-f8f8-4204-ba6e-ffaeae7d3649)
Jun 08 21:38:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: a841b032-545a-476b-a7ea-74383319a60d
Jun 08 21:38:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [67923db84c7e48c2a31978952c4bf44f]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (a841b032-545a-476b-a7ea-74383319a60d)
Jun 08 21:38:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 3da2800a-10e0-4c4c-a106-64ff4594be5a
Jun 08 21:38:11 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (3da2800a-10e0-4c4c-a106-64ff4594be5a)
Jun 08 21:44:53 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1327@foreman.test01.com
Jun 08 21:46:10 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1331@foreman.test01.com
Jun 08 21:48:16 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 1330@foreman.test01.com
Jun 08 21:49:55 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: dc385622-7c3b-4472-a751-c03ea4d3822b
Jun 08 21:49:55 foreman.test01.com pulpcore-worker-4[1327]: pulp [2bb9d0449bdd49eabcc3745f8322717e]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (dc385622-7c3b-4472-a751-c03ea4d3822b)
Jun 08 21:49:55 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 245f9d6d-21c6-400f-9720-0f8249c4b5f3
Jun 08 21:49:55 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (245f9d6d-21c6-400f-9720-0f8249c4b5f3)
Jun 08 21:49:55 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 37bbe9e0-ed02-4978-a9ce-62d06641c102
Jun 08 21:49:56 foreman.test01.com pulpcore-worker-4[1327]: pulp [e3087f6582fc46718f476e653929f7e9]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (37bbe9e0-ed02-4978-a9ce-62d06641c102)
Jun 08 21:49:56 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 3165eea2-86e8-44b0-bed7-af4fe0debfc3
Jun 08 21:49:56 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (3165eea2-86e8-44b0-bed7-af4fe0debfc3)
Jun 08 21:51:00 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 56ede754-8558-4945-8abd-28fe6b10ce4e
Jun 08 21:51:00 foreman.test01.com pulpcore-worker-4[1327]: pulp [7239aa285126484c8d4fbc135268b5bd]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (56ede754-8558-4945-8abd-28fe6b10ce4e)
Jun 08 21:51:00 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 4b2e7103-1a88-4454-8bd8-2ed97965f1b2
Jun 08 21:51:01 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (4b2e7103-1a88-4454-8bd8-2ed97965f1b2)
Jun 08 21:51:01 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: a695c5a4-cc32-4bbe-a9fd-ff98120d765b
Jun 08 21:51:01 foreman.test01.com pulpcore-worker-4[1327]: pulp [886d3c7bfa9d4e069d72f0da2af9ff44]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (a695c5a4-cc32-4bbe-a9fd-ff98120d765b)
Jun 08 21:51:01 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: 159f46be-2ee5-4e83-b841-856de5eaa202
Jun 08 21:51:01 foreman.test01.com pulpcore-worker-4[1327]: pulp [None]: rq.worker:INFO: 1327@foreman.test01.com: Job OK (159f46be-2ee5-4e83-b841-856de5eaa202)
Jun 08 21:52:50 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: e37618e1-947b-4050-af2d-62a43da00893
Jun 08 21:52:50 foreman.test01.com pulpcore-worker-3[1330]: pulp [fa3f168413f34502a87c1c25f1f5398e]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (e37618e1-947b-4050-af2d-62a43da00893)
Jun 08 21:52:51 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: 50fc3d6a-3a2a-4c26-91a5-6910ec849200
Jun 08 21:52:51 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (50fc3d6a-3a2a-4c26-91a5-6910ec849200)
Jun 08 21:52:51 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: a4212b4f-0d5d-4f06-9548-845054fb573b
Jun 08 21:52:52 foreman.test01.com pulpcore-worker-3[1330]: pulp [8cd84aedfd71440f847cd88025bdd086]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (a4212b4f-0d5d-4f06-9548-845054fb573b)
Jun 08 21:52:52 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: f4b3b4cf-860d-428b-8a67-54be2adcfcd3
Jun 08 21:52:53 foreman.test01.com pulpcore-worker-3[1330]: pulp [None]: rq.worker:INFO: 1330@foreman.test01.com: Job OK (f4b3b4cf-860d-428b-8a67-54be2adcfcd3)
Jun 08 21:53:20 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: 490ce10c-e37c-43bc-85a1-f83e7cadef2a
Jun 08 21:53:25 foreman.test01.com pulpcore-worker-1[1331]: pulp [6a686b0a4c494525bfcc36b61cd07331]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (490ce10c-e37c-43bc-85a1-f83e7cadef2a)
Jun 08 21:53:25 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: 9f67d9fe-92ab-4f0c-8bf7-f39eb13496fa
Jun 08 21:53:25 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (9f67d9fe-92ab-4f0c-8bf7-f39eb13496fa)
Jun 08 21:53:25 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: de325b4d-b8a3-44bf-9243-fcc073b63099
Jun 08 21:53:26 foreman.test01.com pulpcore-worker-1[1331]: pulp [dfdf6930f5e44803bfb7ed8e3a803c3a]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (de325b4d-b8a3-44bf-9243-fcc073b63099)
Jun 08 21:53:26 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: a58aabdc-2b3a-4e1c-aa2f-2bc97ddbef5c
Jun 08 21:53:26 foreman.test01.com pulpcore-worker-1[1331]: pulp [None]: rq.worker:INFO: 1331@foreman.test01.com: Job OK (a58aabdc-2b3a-4e1c-aa2f-2bc97ddbef5c)
Jun 08 21:54:32 foreman.test01.com pulpcore-worker-2[1332]: pulp [None]: rq.worker:INFO: 1332@foreman.test01.com: 2b5f7df6-3831-41cd-9aaf-019d09757959
Jun 08 21:54:32 foreman.test01.com pulpcore-worker-2[1332]: pulp [8d3b76f15f5043639ca8b7e032c64b23]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=RHEL-7-REPO-131233 remote=RHEL-7-REPO-137732

Ahh, so you are hitting a couple of issues:

  1. syncing is still failing with ‘response payload is not completed’

I’ll check in with the pulp team about this

  1. your content view publish is failing with the proxy error

would try reducing the ‘bulk load size’ in katello, essentially katello is fetching a lot of items and pulp is not responding within the timeout time. We can adjust both sides of this to try to improve it.

in /etc/foreman/plugins/katello.yaml

add under :katello_applicability: true

  :katello_applicability: true
  :pulp:
    :bulk_load_size: 500

and restart the services:

foreman-maintain service restart

then give it a try. Note that this will be wiped out anytime you re-run the installer, but we’ve made it easier in 4.1 to adjust this setting from the Administer > settings page in 4.1

and resume your CV publish

Thanks for your guideline, i’m following your instruction. But i didn’t properly understand your last line.

after you’ve made the change, click the ‘resume’ button on the failed content view publish task.

Also, could you run this command:

 sudo -u postgres psql pulpcore -c 'select url, download_concurrency from core_remote;'

and provide the output? I’d like to confirm that the download_concurrency was updated properly. Thanks!

download

That doesn’t look like it got updated. Its still set to 10 by default. Did you run the hammer repository update command? Did it error?

No i didn’t run hammer repository update till now. But after made change in katello.yml this time repo sync didn’t stop, it is running currently.

If i face same issue like as before i will definitely execute hammer repository update.

oh! Well that was meant to solve your CV Publish error.

The hammer repository update command is meant to solve your sync error. Different solutions for different problems :slight_smile:

1 Like

Ok i got it :grinning:. Now let me check if the sync error persists or not after execute hammer repository update command.