Awesome! Glad I could help
1 Like
So, I’m getting something very similar looking on 2 Oracle Linux 8 repositories, with Katello 4.2.0/pulp-core 3.14.8/pulp-rpm 3.14.6.
The ‘baseos’ and ‘appstream’ repositories won’t sync, and I can’t publish new CVVs for the CV containing these repositories either.
Which version of Katello should fix this?
So, I found https://github.com/Katello/katello/commit/a90925ac3dae880a0cd8eb52e2584babbaab1b30 - I presume that means the answer is Katello 4.2.1?
I’ve been watching this too. 4.2.1 is in the release process now (Katello 4.2.1 release process). I am hopeful this fix will be included…
Running 4,2,1, I seem to be seeing the same error:
Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Tue, 09 Nov 2021 03:54:19 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"12599", "correlation-id"=>"0912d652-dcb0-46f6-b68e-6f221f94477f", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 srv-katello.imladris.lan", "connection"=>"close"}
Response body: ["Could not find the following content units: ['/pulp/api/v3/content/rpm/advisories/ceaa4290-488c-42cb-9f3a-b9d2da2ffc40/', '/pulp/api/v3/content/rpm/advisories/a6d3a3fb-bc29-4f55-b7a9-e38ba7fa8515/', '/pulp/api/v3/content/rpm/advisories/61443fa5-34a6-4775-a64a-4cad51b4ffcf/', '/pulp/api/v3/content/rpm/advisories/763658ee-c224-43fe-8026-cfdd7b372d8b/', '/pulp/api/v3/content/rpm/advisories/800d94b3-f516-409f-a662-71c3d1dd4ed1/', '/pulp/api/v3/content/rpm/advisories/6a8f8ed6-7b5e-455e-a1f4-56c2155442c7/', '/pulp/api/v3/content/rpm/advisories/a42f9b4c-af16-401d-988b-935b4acdf920/', '/pulp/api/v3/content/rpm/advisories/91225f5c-7f7f-4d40-9319-6cf45b1ae814/', '/pulp/api/v3/content/rpm/advisories/e4492df0-eeaf-441d-a63e-c4aa663f95f9/', '/pulp/api/v3/content/rpm/advisories/36afe4bc-d04c-4a5d-91fa-ba400ce6a0ac/', '/pulp/api/v3/content/rpm/advisories/283205ec-12f9-4b16-9830-e954b999c538/', '/pulp/api/v3/content/rpm/advisories/5975bb24-9b6a-4076-9dce-ce98aa210c40/', '/pulp/api/v3/content/rpm/advisories/b54b6138-06ee-423e-aff6-0bafb4c1c977/', '/pulp/api/v3/content/rpm/advisories/239a2bff-6c62-4a2c-b306-84322d9c0060/', '/pulp/api/v3/content/rpm/advisories/f7206432-61b2-4da6-85e3-be2058f03840/', '/pulp/api/v3/content/rpm/advisories/1a60b4ae-03b1-4519-b5f6-98ac1d55a844/', '/pulp/api/v3/content/rpm/advisories/e64024f6-188e-46b9-85ab-c52ece9ed146/', '/pulp/api/v3/content/rpm/advisories/aae4a3cd-2301-4955-a7be-9ae78177ad7e/', '/pulp/api/v3/content/rpm/advisories/57914258-eab1-44fa-b052-d6a1ad0185ff/', '/pulp/api/v3/content/rpm/advisories/0990ab7c-09e4-46b6-8988-77bb6ae15751/', '/pulp/api/v3/content/rpm/advisories/695f2959-9e44-44d7-98f9-df4ab3173331/', '/pulp/api/v3/content/rpm/advisories/806e22b9-7619-4b22-9daf-e80f0cbf25b1/', '/pulp/api/v3/content/rpm/advisories/33a95496-e251-466c-9660-b6eecf128d50/', '/pulp/api/v3/content/rpm/advisories/f568a850-e715-420b-be5d-5d373dd36c8e/', '/pulp/api/v3/content/rpm/advisories/e6612c11-07e3-418c-bc49-3d9d36b55fc0/', '/pulp/api/v3/content/rpm/advisories/d6e087bc-eb6e-4807-b9c5-bc31adb93a5a/', '/pulp/api/v3/content/rpm/advisories/5bf1b4e1-6ade-42d5-95c8-72744920c93a/', '/pulp/api/v3/content/rpm/advisories/fb3d2f44-488c-4b2c-8372-808951d51030/', '/pulp/api/v3/content/rpm/advisories/a54274fa-3925-4ebd-a880-ccf167e64497/', '/pulp/api/v3/content/rpm/advisories/fd30be41-b6f1-4d6a-872b-ccfa36e5ff7a/', '/pulp/api/v3/content/rpm/advisories/724d4d15-2fae-48b9-a28c-441e8dc46f15/', '/pulp/api/v3/content/rpm/advisories/ac8f4922-b292-4c46-80c3-ab7ba3a7da2a/', '/pulp/api/v3/content/rpm/advisories/d8ae7cb3-327a-4464-a8ae-8ec312e75c65/', '/pulp/api/v3/content/rpm/advisories/ff7897ef-bffd-488e-8d09-5b9233c47bb2/', '/pulp/api/v3/content/rpm/advisories/6fa64241-1fba-490d-9d20-b979d9a70beb/', '/pulp/api/v3/content/rpm/advisories/6d4a65da-082f-4887-aed7-7570142d89df/', '/pulp/api/v3/content/rpm/advisories/51a26cb2-5905-409b-9dec-3c494483c524/', '/pulp/api/v3/content/rpm/advisories/85306bd3-943e-4c74-9f83-ce83801b2dc8/', '/pulp/api/v3/content/rpm/advisories/532630a0-8511-4a82-8ae9-b03ccd891b7a/', '/pulp/api/v3/content/rpm/advisories/018753b1-320a-423b-a4eb-e4394ecd1d60/', '/pulp/api/v3/content/rpm/advisories/a6836381-9054-4a0c-8e83-e64e97decbc4/', '/pulp/api/v3/content/rpm/advisories/a229daa5-dcc9-415b-8766-d90d376cc424/', '/pulp/api/v3/content/rpm/advisories/8de22076-e4f2-4d95-be64-5f0a6dc967e8/', '/pulp/api/v3/content/rpm/advisories/bb8f04ae-b5e9-4a2a-83a8-e72c9dfe523f/', '/pulp/api/v3/content/rpm/advisories/8a1aa5dc-74f9-48e7-8100-b34e0f3a7605/', '/pulp/api/v3/content/rpm/advisories/edaa5a7e-c373-412b-9d1c-9341c2dff1dd/', '/pulp/api/v3/content/rpm/advisories/340d1d11-e52f-43d4-97e9-e45fb181e7f4/', '/pulp/api/v3/content/rpm/advisories/66858017-bcf5-436e-9ea8-f84b5556e90c/', '/pulp/api/v3/content/rpm/advisories/40365fcf-ce5b-41a6-bec0-97278baeaccc/', '/pulp/api/v3/content/rpm/advisories/add5d26d-458a-41a0-9fc4-311682ed04c9/', '/pulp/api/v3/content/rpm/advisories/5c9b45bb-5bf5-4cc0-9fcf-0aba9202f3c7/', '/pulp/api/v3/content/rpm/advisories/48cd6129-10ab-40e7-ab56-1e040d18e879/', '/pulp/api/v3/content/rpm/advisories/194331bd-70fe-4390-881d-65326e999c72/', '/pulp/api/v3/content/rpm/advisories/6a447d71-4b80-4faa-b614-3cb6ba79a114/', '/pulp/api/v3/content/rpm/advisories/8dd072b5-1ae5-4671-b059-ffe401511b3f/', '/pulp/api/v3/content/rpm/advisories/3a7c9c38-26e5-401b-a770-eebe9348bd47/', '/pulp/api/v3/content/rpm/advisories/17d5b4ec-4e35-49b0-8d23-e794336bc820/', '/pulp/api/v3/content/rpm/advisories/ba8ea5d5-7b2f-47e5-ab54-6aaac47d4414/', '/pulp/api/v3/content/rpm/advisories/c97ef57e-b0bc-4868-8db6-91cf0ef51289/', '/pulp/api/v3/content/rpm/advisories/6de042b9-7eb3-485c-a49b-b7511d742ec6/', '/pulp/api/v3/content/rpm/advisories/38f44bdf-934e-4930-97f5-012b43ce28ca/', '/pulp/api/v3/content/rpm/advisories/b99ac40e-ce64-4738-a14e-f496866257d1/', '/pulp/api/v3/content/rpm/advisories/ad06de0c-b376-4005-a3d6-2fae2d98a916/', '/pulp/api/v3/content/rpm/advisories/5340e7c6-2b3e-4a89-9376-bd3117555222/', '/pulp/api/v3/content/rpm/advisories/a6513de5-7158-4aaa-aa40-60ddb9047810/', '/pulp/api/v3/content/rpm/advisories/899a8b6c-dd0e-4318-92b3-cb5307e0d5f0/', '/pulp/api/v3/content/rpm/advisories/8924917f-4a32-41a0-8b85-7bd7f3a481aa/', '/pulp/api/v3/content/rpm/advisories/8b5196fe-2846-4250-9bec-449306fd3efd/', '/pulp/api/v3/content/rpm/advisories/943057b0-76ac-4509-9499-ec779029d133/', '/pulp/api/v3/content/rpm/advisories/4d7b6910-df4b-4876-bf43-f3c66af01fe8/', '/pulp/api/v3/content/rpm/advisories/65f5048e-b040-4372-8e56-afe31a6031a4/', '/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/', '/pulp/api/v3/content/rpm/advisories/4470e762-2852-47cd-96a3-0ecbdb66a482/', '/pulp/api/v3/content/rpm/advisories/154511ee-0219-420e-b7f0-92bef6c3b9a3/', '/pulp/api/v3/content/rpm/advisories/ec33b0be-64b8-4530-888b-950aab0d9b3c/', '/pulp/api/v3/content/rpm/advisories/cf8b2756-6c9e-466f-9c34-c8a7f33cee05/', '/pulp/api/v3/content/rpm/advisories/91a14008-544d-4ca7-9901-10caaa5c7fb6/', '/pulp/api/v3/content/rpm/advisories/2419b15b-2fd2-4fb9-b9f9-574eb08877c0/', '/pulp/api/v3/content/rpm/advisories/3ff0539f-93aa-4f77-ada6-b49c0decdae8/', '/pulp/api/v3/content/rpm/advisories/8bdbe6a3-2402-40f5-90a5-a768acf20489/', '/pulp/api/v3/content/rpm/advisories/df9a469d-8925-4777-a9e2-17fde145d2fc/', '/pulp/api/v3/content/rpm/advisories/79b0cb71-bad1-42d9-9f2e-d2bb4370a193/', '/pulp/api/v3/content/rpm/advisories/3e515cbb-15a7-4886-a5b5-d06b79bc6fda/', '/pulp/api/v3/content/rpm/advisories/a7105213-a985-48bf-a30c-06782b99edae/', '/pulp/api/v3/content/rpm/advisories/c884c08b-2f57-489c-98ae-441e989f491c/', '/pulp/api/v3/content/rpm/advisories/24a3d212-0ba7-495e-92f0-d24ec18c26c4/', '/pulp/api/v3/content/rpm/advisories/c9da9b79-d831-480c-b23c-7d955ed7d1de/', '/pulp/api/v3/content/rpm/advisories/5816e295-b425-40da-bce2-996e0590f572/', '/pulp/api/v3/content/rpm/advisories/66cee998-68af-4d2b-972d-a8bc88395cec/', '/pulp/api/v3/content/rpm/advisories/2d57e016-8d87-4cd3-ae2d-40597e67a1e3/', '/pulp/api/v3/content/rpm/advisories/418fcc4c-4cba-4962-84e9-beade6d87297/', '/pulp/api/v3/content/rpm/advisories/11ded541-d6ad-4312-8d12-3746a0b5849a/', '/pulp/api/v3/content/rpm/advisories/9cfdb9a3-ba08-47dd-b538-023c65e9795c/', '/pulp/api/v3/content/rpm/advisories/b2334511-27a5-46e3-890e-9ddca1c6a974/', '/pulp/api/v3/content/rpm/advisories/be19e56e-d1ab-443a-9479-972d832d33d4/', '/pulp/api/v3/content/rpm/advisories/633f337e-a0f0-4c6b-ad0d-b6fdbfe470f7/', '/pulp/api/v3/content/rpm/advisories/8238cc1b-6f4b-428f-8d5e-37bca6969624/', '/pulp/api/v3/content/rpm/advisories/4566ee58-c015-4551-823b-37069f2ebfa9/', '/pulp/api/v3/content/rpm/advisories/0cbfbba5-5edd-4376-9bce-0bf38b9e4aee/', '/pulp/api/v3/content/rpm/advisories/d743e1de-3da3-4b49-ab53-d7fa3b74e3b5/', '/pulp/api/v3/content/rpm/advisories/f7378c53-b1cb-473b-9be6-73ba20341788/', '/pulp/api/v3/content/rpm/advisories/6ada50f6-24b5-45aa-bbd1-4bc82abb6752/', '/pulp/api/v3/content/rpm/advisories/7027ca2d-3dd8-439e-8d6e-66bf47de46e2/', '/pulp/api/v3/content/rpm/advisories/5ab1679c-e4a7-46a4-9869-15a4230a09f0/', '/pulp/api/v3/content/rpm/advisories/81a396a0-c69e-4990-955e-e23f2654516a/', '/pulp/api/v3/content/rpm/advisories/a3b907b1-3446-4a46-9399-2e6e02d65b9d/', '/pulp/api/v3/content/rpm/advisories/12f53e42-bcd5-45c4-b66c-36beea997280/', '/pulp/api/v3/content/rpm/advisories/0dc867a9-60f6-4f7e-b928-bc5171776120/', '/pulp/api/v3/content/rpm/advisories/e7eebc32-7ad2-4121-85db-4e4ff76b4c5c/', '/pulp/api/v3/content/rpm/advisories/a9f59032-3061-47f5-b7b8-b2af1c541385/', '/pulp/api/v3/content/rpm/advisories/803791f0-7891-41c6-8315-f4965ff4aa05/', '/pulp/api/v3/content/rpm/advisories/ff259aac-f67e-4288-8ece-39ac4b63c618/', '/pulp/api/v3/content/rpm/advisories/c747ef5c-d21c-4c4d-b0df-6048ad046d31/', '/pulp/api/v3/content/rpm/advisories/f6dbe489-d5f1-4a5b-a09e-958222aa02e8/', '/pulp/api/v3/content/rpm/advisories/85b49054-83d8-40ac-b4c6-85a85c53ae7b/', '/pulp/api/v3/content/rpm/advisories/d2d63b60-3ddb-4793-acb9-0bc2d8e94de7/', '/pulp/api/v3/content/rpm/advisories/f96c30ed-ab9f-43af-bc18-2096d87a94bb/', '/pulp/api/v3/content/rpm/advisories/47507859-8076-4ff1-ba4d-cdff8c7c53b7/', '/pulp/api/v3/content/rpm/advisories/043bd7f6-cba9-46a5-91f4-9f22a1effe90/', '/pulp/api/v3/content/rpm/advisories/a6be09c7-8fae-49e1-80dc-1d5f9165e94b/', '/pulp/api/v3/content/rpm/advisories/e25ac705-c7ea-4865-a80b-e4581fafa660/', '/pulp/api/v3/content/rpm/advisories/5daed36c-2698-41af-9b47-4b4a6def4094/', '/pulp/api/v3/content/rpm/advisories/882d3c55-a3ef-4bcc-8748-030e697e0d65/', '/pulp/api/v3/content/rpm/advisories/a8d14775-5a1a-4176-bb91-101397dacf72/', '/pulp/api/v3/content/rpm/advisories/c9fde76e-3def-4c0b-bdcf-949dceb52479/', '/pulp/api/v3/content/rpm/advisories/04d69f3e-9a3c-4845-a2ac-24a8b350eb38/', '/pulp/api/v3/content/rpm/advisories/207d5736-126f-45bf-939b-4dc89d5b57fd/', '/pulp/api/v3/content/rpm/advisories/2fcc7fa1-d14c-4261-8b70-c3d1bdc313b4/', '/pulp/api/v3/content/rpm/advisories/8d1f38f5-3b0d-48d5-a4e1-75957d7d64be/', '/pulp/api/v3/content/rpm/advisories/a06a5caf-0244-4738-8c01-5246199128d4/', '/pulp/api/v3/content/rpm/advisories/3db0905e-99a3-4015-b23f-e01a3ca02116/', '/pulp/api/v3/content/rpm/advisories/c9924d6a-587d-45ce-a5e4-e9ebac238f8b/', '/pulp/api/v3/content/rpm/advisories/705c5624-1848-40fc-b5c4-5da6120d0fbd/', '/pulp/api/v3/content/rpm/advisories/a037f797-6754-47c8-ac79-b299a46a1305/', '/pulp/api/v3/content/rpm/advisories/3d55da91-a081-449c-b8d2-4a539a505c4c/', '/pulp/api/v3/content/rpm/advisories/8477b0d4-8fbc-4d75-bf0f-f869a0183ea2/', '/pulp/api/v3/content/rpm/advisories/035d4617-b796-40d6-af41-ad19556bc17c/', '/pulp/api/v3/content/rpm/advisories/a59dcb2b-1b0e-4b10-811b-00a94fc7d4c3/', '/pulp/api/v3/content/rpm/advisories/22bce75c-3f3a-4cc2-a1e5-cca2bfd7ef48/', '/pulp/api/v3/content/rpm/advisories/aeb89d15-10ab-40e7-ac9b-d72816949fe5/', '/pulp/api/v3/content/rpm/advisories/b1d37fbc-2db2-4d3b-9ddf-2fc3b56f4500/', '/pulp/api/v3/content/rpm/advisories/5cf0a49e-f276-46f7-9f81-4ecef4b84f23/', '/pulp/api/v3/content/rpm/advisories/7dc21dae-7b88-4998-b60f-b4d12cc0e32e/', '/pulp/api/v3/content/rpm/advisories/9ec9e087-eb98-4052-bba3-cdcaceabe509/', '/pulp/api/v3/content/rpm/advisories/242f4ac3-2e69-42da-99be-75544cc7b546/', '/pulp/api/v3/content/rpm/advisories/4c7043a2-a93f-4320-87cc-fa4e3cb62141/', '/pulp/api/v3/content/rpm/advisories/36cc757a-f903-458c-8836-c000bd7e7716/', '/pulp/api/v3/content/rpm/advisories/bdae9e29-2f2c-4018-8d2c-4017983f9536/', '/pulp/api/v3/content/rpm/advisories/b48aaea6-89cc-4a97-9b90-92820766aad9/', '/pulp/api/v3/content/rpm/advisories/56a80ec2-d7f9-458c-9db2-110bee0d8c96/', '/pulp/api/v3/content/rpm/advisories/d0050502-44fd-4985-b62f-beadc11ae21c/', '/pulp/api/v3/content/rpm/advisories/599e04cf-116b-4836-b6cd-c8fc582de3bb/', '/pulp/api/v3/content/rpm/advisories/b88a9079-b5bc-4407-bb32-18aa429a04fb/', '/pulp/api/v3/content/rpm/advisories/4a954fd4-4537-43dd-abdc-80596ae97387/', '/pulp/api/v3/content/rpm/advisories/5caacf44-d4b6-482e-bcc6-56cf7005f7db/', '/pulp/api/v3/content/rpm/advisories/ee53dcb5-6e2c-4ba8-8014-d12c40571000/', '/pulp/api/v3/content/rpm/advisories/c350e7ea-726c-4332-900a-14f990c26b7b/', '/pulp/api/v3/content/rpm/advisories/0596f80f-6a7d-4e24-bd00-2875a9e9b5ae/', '/pulp/api/v3/content/rpm/advisories/d767cbab-b708-43c1-973e-5a3fad0140f6/', '/pulp/api/v3/content/rpm/advisories/f5244a1b-dc24-40eb-8adf-ba7dc5f9d5c8/', '/pulp/api/v3/content/rpm/advisories/c7af66e7-f80f-4454-9776-4cd411646194/', '/pulp/api/v3/content/rpm/advisories/a9449704-2ee1-4d81-9131-791c6abda6c7/', '/pulp/api/v3/content/rpm/advisories/590b37a6-9f3f-42dc-be26-5f762b32e30a/']"]
So, updated to Katello 4.2.1, which seems to have fixed one class of sync error, but now getting a different one for one Oracle Linux repository (codeready-builder):
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: pulp [e3bdacb4-16b7-4a71-b7fd-2d7d33bdaf27]: pulpcore.tasking.pulpcore_worker:INFO: Task f3ad9aac-7908-42c5-afc7-dc239390e0be failed ([Errno 2] No such file or directory: './tmpfjusco_9')
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: pulp [e3bdacb4-16b7-4a71-b7fd-2d7d33bdaf27]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: result = func(*args, **kwargs)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 490, in synchronize
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: version = dv.create()
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: loop.run_until_complete(pipeline)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib64/python3.6/tempfile.py", line 809, in __exit__
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: self.cleanup()
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib64/python3.6/tempfile.py", line 813, in cleanup
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: _shutil.rmtree(self.name)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib64/python3.6/shutil.py", line 477, in rmtree
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: onerror(os.lstat, path, sys.exc_info())
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib64/python3.6/shutil.py", line 475, in rmtree
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: orig_st = os.lstat(path)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: Traceback (most recent call last):
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/bin/pulpcore-worker", line 11, in <module>
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: return self.main(*args, **kwargs)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: rv = self.invoke(ctx)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: return ctx.invoke(self.callback, **ctx.params)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: return callback(*args, **kwargs)
Nov 11 12:07:28 katello.example.com pulpcore-worker-2[2120366]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
May have spoken too soon, latest sync of OL8 appstream repo is now throwing:
Pulp task errorError message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 11 Nov 2021 12:27:12 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "correlation-id"=>"99632976-4bb5-4595-9af1-cc488c974cef", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 katello.example.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}
Digging further into the logs, it’s still the following, as described in Issue #9503: get() returned more than one UpdateRecord -- it returned X! - RPM Support - Pulp
Nov 11 12:31:07 katello.example.com pulpcore-worker-3[2130750]: pulp [557cd175-352f-4da1-ad93-b19852ac0436]: pulpcore.tasking.pulpcore_worker:INFO: Task fe898536-ef36-4eb1-8c39-c6365dc5ab96 failed (get() returned more than one UpdateRecord -- it returned 2!)
Nov 11 12:31:07 katello.example.com pulpcore-worker-3[2130750]: pulp [557cd175-352f-4da1-ad93-b19852ac0436]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
Oh, just to clarify - this looks to be a different repository and failure condition. The appstream sync is still running, even after the “Invalid hyperlink” error was thrown.
@mhjacks and @John_Beranek,
To fix the original issue in this thread, once you’re on 4.2.1, try a complete sync of the repositories that are in the “bad” content view versions. Then, republish. Let me know if you’re still having the error.
@John_Beranek if you’re still having sync issues, do you mind opening up a new thread? The errors you’ve posted look familiar.
@iballou thanks for the advice.
I have done complete sync of the two repositories in the “bad” content view - and I’m still seeing the error on content view publish.
@mhjacks gotcha, we’ll probably need to do some database cleanup then. Tomorrow I can recreate the issue and find a more generalized solution.
I believe the issue is that your database has errata-repository relation records in it that were orphaned by the bug.
1 Like
Trying a complete sync of both repositories now.
OK, no change on a full sync. Both synchronisations have ended in a warning state, and sent out the “Sync summary” email.
1 Like
I’m thinking we might need to run complete sync on more repositories. On my system, a complete sync fixed the records pointing to non-existing errata. Can you show me the output of the following?
::Katello::Repository.where(id: ::Katello::RepositoryErratum.where(erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/")).map{|r| r.root.name}
If you haven’t run a complete sync on any of the repositories with matching names that pop up from that command, please run it on them as well and then try the publish again. Let me know if you’ve synced them all already.
If you’re still seeing the 09d8fae6-4043-4809-9624-a19e6a440332 erratum/advisory in the error, we may need to clean it manually. If the error pops up but that erratum isn’t in there, then other repositories may also require a complete sync.
OK, I’m not entirely sure what I’m looking at here. I’ve currently got this error message on the failed promotion:
HTTP status code: 400
Response headers: {"date"=>"Fri, 12 Nov 2021 20:03:04 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"10828", "correlation-id"=>"11610a41-26f6-4de7-b006-59f931ebc004", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 srv-katello.imladris.lan", "connection"=>"close"}
Response body: ["Could not find the following content units: ['/pulp/api/v3/content/rpm/advisories/633f337e-a0f0-4c6b-ad0d-b6fdbfe470f7/', '/pulp/api/v3/content/rpm/advisories/4566ee58-c015-4551-823b-37069f2ebfa9/', '/pulp/api/v3/content/rpm/advisories/add5d26d-458a-41a0-9fc4-311682ed04c9/', '/pulp/api/v3/content/rpm/advisories/194331bd-70fe-4390-881d-65326e999c72/', '/pulp/api/v3/content/rpm/advisories/8924917f-4a32-41a0-8b85-7bd7f3a481aa/', '/pulp/api/v3/content/rpm/advisories/a6be09c7-8fae-49e1-80dc-1d5f9165e94b/', '/pulp/api/v3/content/rpm/advisories/c884c08b-2f57-489c-98ae-441e989f491c/', '/pulp/api/v3/content/rpm/advisories/40365fcf-ce5b-41a6-bec0-97278baeaccc/', '/pulp/api/v3/content/rpm/advisories/5caacf44-d4b6-482e-bcc6-56cf7005f7db/', '/pulp/api/v3/content/rpm/advisories/6a8f8ed6-7b5e-455e-a1f4-56c2155442c7/', '/pulp/api/v3/content/rpm/advisories/4470e762-2852-47cd-96a3-0ecbdb66a482/', '/pulp/api/v3/content/rpm/advisories/8de22076-e4f2-4d95-be64-5f0a6dc967e8/', '/pulp/api/v3/content/rpm/advisories/724d4d15-2fae-48b9-a28c-441e8dc46f15/', '/pulp/api/v3/content/rpm/advisories/d767cbab-b708-43c1-973e-5a3fad0140f6/', '/pulp/api/v3/content/rpm/advisories/4a954fd4-4537-43dd-abdc-80596ae97387/', '/pulp/api/v3/content/rpm/advisories/38f44bdf-934e-4930-97f5-012b43ce28ca/', '/pulp/api/v3/content/rpm/advisories/ba8ea5d5-7b2f-47e5-ab54-6aaac47d4414/', '/pulp/api/v3/content/rpm/advisories/2419b15b-2fd2-4fb9-b9f9-574eb08877c0/', '/pulp/api/v3/content/rpm/advisories/a6d3a3fb-bc29-4f55-b7a9-e38ba7fa8515/', '/pulp/api/v3/content/rpm/advisories/3e515cbb-15a7-4886-a5b5-d06b79bc6fda/', '/pulp/api/v3/content/rpm/advisories/17d5b4ec-4e35-49b0-8d23-e794336bc820/', '/pulp/api/v3/content/rpm/advisories/0cbfbba5-5edd-4376-9bce-0bf38b9e4aee/', '/pulp/api/v3/content/rpm/advisories/66858017-bcf5-436e-9ea8-f84b5556e90c/', '/pulp/api/v3/content/rpm/advisories/47507859-8076-4ff1-ba4d-cdff8c7c53b7/', '/pulp/api/v3/content/rpm/advisories/f7378c53-b1cb-473b-9be6-73ba20341788/', '/pulp/api/v3/content/rpm/advisories/edaa5a7e-c373-412b-9d1c-9341c2dff1dd/', '/pulp/api/v3/content/rpm/advisories/d6e087bc-eb6e-4807-b9c5-bc31adb93a5a/', '/pulp/api/v3/content/rpm/advisories/f5244a1b-dc24-40eb-8adf-ba7dc5f9d5c8/', '/pulp/api/v3/content/rpm/advisories/705c5624-1848-40fc-b5c4-5da6120d0fbd/', '/pulp/api/v3/content/rpm/advisories/f96c30ed-ab9f-43af-bc18-2096d87a94bb/', '/pulp/api/v3/content/rpm/advisories/48cd6129-10ab-40e7-ab56-1e040d18e879/', '/pulp/api/v3/content/rpm/advisories/aeb89d15-10ab-40e7-ac9b-d72816949fe5/', '/pulp/api/v3/content/rpm/advisories/599e04cf-116b-4836-b6cd-c8fc582de3bb/', '/pulp/api/v3/content/rpm/advisories/9cfdb9a3-ba08-47dd-b538-023c65e9795c/', '/pulp/api/v3/content/rpm/advisories/803791f0-7891-41c6-8315-f4965ff4aa05/', '/pulp/api/v3/content/rpm/advisories/899a8b6c-dd0e-4318-92b3-cb5307e0d5f0/', '/pulp/api/v3/content/rpm/advisories/c9da9b79-d831-480c-b23c-7d955ed7d1de/', '/pulp/api/v3/content/rpm/advisories/035d4617-b796-40d6-af41-ad19556bc17c/', '/pulp/api/v3/content/rpm/advisories/943057b0-76ac-4509-9499-ec779029d133/', '/pulp/api/v3/content/rpm/advisories/4d7b6910-df4b-4876-bf43-f3c66af01fe8/', '/pulp/api/v3/content/rpm/advisories/a59dcb2b-1b0e-4b10-811b-00a94fc7d4c3/', '/pulp/api/v3/content/rpm/advisories/9ec9e087-eb98-4052-bba3-cdcaceabe509/', '/pulp/api/v3/content/rpm/advisories/be19e56e-d1ab-443a-9479-972d832d33d4/', '/pulp/api/v3/content/rpm/advisories/b99ac40e-ce64-4738-a14e-f496866257d1/', '/pulp/api/v3/content/rpm/advisories/4c7043a2-a93f-4320-87cc-fa4e3cb62141/', '/pulp/api/v3/content/rpm/advisories/800d94b3-f516-409f-a662-71c3d1dd4ed1/', '/pulp/api/v3/content/rpm/advisories/df9a469d-8925-4777-a9e2-17fde145d2fc/', '/pulp/api/v3/content/rpm/advisories/0990ab7c-09e4-46b6-8988-77bb6ae15751/', '/pulp/api/v3/content/rpm/advisories/8bdbe6a3-2402-40f5-90a5-a768acf20489/', '/pulp/api/v3/content/rpm/advisories/5340e7c6-2b3e-4a89-9376-bd3117555222/', '/pulp/api/v3/content/rpm/advisories/763658ee-c224-43fe-8026-cfdd7b372d8b/', '/pulp/api/v3/content/rpm/advisories/85b49054-83d8-40ac-b4c6-85a85c53ae7b/', '/pulp/api/v3/content/rpm/advisories/a037f797-6754-47c8-ac79-b299a46a1305/', '/pulp/api/v3/content/rpm/advisories/24a3d212-0ba7-495e-92f0-d24ec18c26c4/', '/pulp/api/v3/content/rpm/advisories/8238cc1b-6f4b-428f-8d5e-37bca6969624/', '/pulp/api/v3/content/rpm/advisories/66cee998-68af-4d2b-972d-a8bc88395cec/', '/pulp/api/v3/content/rpm/advisories/c7af66e7-f80f-4454-9776-4cd411646194/', '/pulp/api/v3/content/rpm/advisories/a229daa5-dcc9-415b-8766-d90d376cc424/', '/pulp/api/v3/content/rpm/advisories/6fa64241-1fba-490d-9d20-b979d9a70beb/', '/pulp/api/v3/content/rpm/advisories/85306bd3-943e-4c74-9f83-ce83801b2dc8/', '/pulp/api/v3/content/rpm/advisories/ac8f4922-b292-4c46-80c3-ab7ba3a7da2a/', '/pulp/api/v3/content/rpm/advisories/f7206432-61b2-4da6-85e3-be2058f03840/', '/pulp/api/v3/content/rpm/advisories/c9fde76e-3def-4c0b-bdcf-949dceb52479/', '/pulp/api/v3/content/rpm/advisories/a7105213-a985-48bf-a30c-06782b99edae/', '/pulp/api/v3/content/rpm/advisories/a42f9b4c-af16-401d-988b-935b4acdf920/', '/pulp/api/v3/content/rpm/advisories/3d55da91-a081-449c-b8d2-4a539a505c4c/', '/pulp/api/v3/content/rpm/advisories/5975bb24-9b6a-4076-9dce-ce98aa210c40/', '/pulp/api/v3/content/rpm/advisories/57914258-eab1-44fa-b052-d6a1ad0185ff/', '/pulp/api/v3/content/rpm/advisories/a06a5caf-0244-4738-8c01-5246199128d4/', '/pulp/api/v3/content/rpm/advisories/b54b6138-06ee-423e-aff6-0bafb4c1c977/', '/pulp/api/v3/content/rpm/advisories/8477b0d4-8fbc-4d75-bf0f-f869a0183ea2/', '/pulp/api/v3/content/rpm/advisories/e6612c11-07e3-418c-bc49-3d9d36b55fc0/', '/pulp/api/v3/content/rpm/advisories/a54274fa-3925-4ebd-a880-ccf167e64497/', '/pulp/api/v3/content/rpm/advisories/2fcc7fa1-d14c-4261-8b70-c3d1bdc313b4/', '/pulp/api/v3/content/rpm/advisories/c97ef57e-b0bc-4868-8db6-91cf0ef51289/', '/pulp/api/v3/content/rpm/advisories/a8d14775-5a1a-4176-bb91-101397dacf72/', '/pulp/api/v3/content/rpm/advisories/0596f80f-6a7d-4e24-bd00-2875a9e9b5ae/', '/pulp/api/v3/content/rpm/advisories/ec33b0be-64b8-4530-888b-950aab0d9b3c/', '/pulp/api/v3/content/rpm/advisories/e25ac705-c7ea-4865-a80b-e4581fafa660/', '/pulp/api/v3/content/rpm/advisories/d8ae7cb3-327a-4464-a8ae-8ec312e75c65/', '/pulp/api/v3/content/rpm/advisories/a6513de5-7158-4aaa-aa40-60ddb9047810/', '/pulp/api/v3/content/rpm/advisories/7027ca2d-3dd8-439e-8d6e-66bf47de46e2/', '/pulp/api/v3/content/rpm/advisories/b88a9079-b5bc-4407-bb32-18aa429a04fb/', '/pulp/api/v3/content/rpm/advisories/590b37a6-9f3f-42dc-be26-5f762b32e30a/', '/pulp/api/v3/content/rpm/advisories/65f5048e-b040-4372-8e56-afe31a6031a4/', '/pulp/api/v3/content/rpm/advisories/bdae9e29-2f2c-4018-8d2c-4017983f9536/', '/pulp/api/v3/content/rpm/advisories/12f53e42-bcd5-45c4-b66c-36beea997280/', '/pulp/api/v3/content/rpm/advisories/695f2959-9e44-44d7-98f9-df4ab3173331/', '/pulp/api/v3/content/rpm/advisories/36afe4bc-d04c-4a5d-91fa-ba400ce6a0ac/', '/pulp/api/v3/content/rpm/advisories/ff259aac-f67e-4288-8ece-39ac4b63c618/', '/pulp/api/v3/content/rpm/advisories/81a396a0-c69e-4990-955e-e23f2654516a/', '/pulp/api/v3/content/rpm/advisories/f6dbe489-d5f1-4a5b-a09e-958222aa02e8/', '/pulp/api/v3/content/rpm/advisories/c9924d6a-587d-45ce-a5e4-e9ebac238f8b/', '/pulp/api/v3/content/rpm/advisories/8a1aa5dc-74f9-48e7-8100-b34e0f3a7605/', '/pulp/api/v3/content/rpm/advisories/5bf1b4e1-6ade-42d5-95c8-72744920c93a/', '/pulp/api/v3/content/rpm/advisories/7dc21dae-7b88-4998-b60f-b4d12cc0e32e/', '/pulp/api/v3/content/rpm/advisories/3ff0539f-93aa-4f77-ada6-b49c0decdae8/', '/pulp/api/v3/content/rpm/advisories/a6836381-9054-4a0c-8e83-e64e97decbc4/', '/pulp/api/v3/content/rpm/advisories/b1d37fbc-2db2-4d3b-9ddf-2fc3b56f4500/', '/pulp/api/v3/content/rpm/advisories/c747ef5c-d21c-4c4d-b0df-6048ad046d31/', '/pulp/api/v3/content/rpm/advisories/018753b1-320a-423b-a4eb-e4394ecd1d60/', '/pulp/api/v3/content/rpm/advisories/3db0905e-99a3-4015-b23f-e01a3ca02116/', '/pulp/api/v3/content/rpm/advisories/d2d63b60-3ddb-4793-acb9-0bc2d8e94de7/', '/pulp/api/v3/content/rpm/advisories/207d5736-126f-45bf-939b-4dc89d5b57fd/', '/pulp/api/v3/content/rpm/advisories/242f4ac3-2e69-42da-99be-75544cc7b546/', '/pulp/api/v3/content/rpm/advisories/33a95496-e251-466c-9660-b6eecf128d50/', '/pulp/api/v3/content/rpm/advisories/340d1d11-e52f-43d4-97e9-e45fb181e7f4/', '/pulp/api/v3/content/rpm/advisories/d743e1de-3da3-4b49-ab53-d7fa3b74e3b5/', '/pulp/api/v3/content/rpm/advisories/61443fa5-34a6-4775-a64a-4cad51b4ffcf/', '/pulp/api/v3/content/rpm/advisories/ff7897ef-bffd-488e-8d09-5b9233c47bb2/', '/pulp/api/v3/content/rpm/advisories/8d1f38f5-3b0d-48d5-a4e1-75957d7d64be/', '/pulp/api/v3/content/rpm/advisories/806e22b9-7619-4b22-9daf-e80f0cbf25b1/', '/pulp/api/v3/content/rpm/advisories/36cc757a-f903-458c-8836-c000bd7e7716/', '/pulp/api/v3/content/rpm/advisories/283205ec-12f9-4b16-9830-e954b999c538/', '/pulp/api/v3/content/rpm/advisories/043bd7f6-cba9-46a5-91f4-9f22a1effe90/', '/pulp/api/v3/content/rpm/advisories/882d3c55-a3ef-4bcc-8748-030e697e0d65/', '/pulp/api/v3/content/rpm/advisories/5c9b45bb-5bf5-4cc0-9fcf-0aba9202f3c7/', '/pulp/api/v3/content/rpm/advisories/b48aaea6-89cc-4a97-9b90-92820766aad9/', '/pulp/api/v3/content/rpm/advisories/5daed36c-2698-41af-9b47-4b4a6def4094/', '/pulp/api/v3/content/rpm/advisories/91225f5c-7f7f-4d40-9319-6cf45b1ae814/', '/pulp/api/v3/content/rpm/advisories/8dd072b5-1ae5-4671-b059-ffe401511b3f/', '/pulp/api/v3/content/rpm/advisories/e7eebc32-7ad2-4121-85db-4e4ff76b4c5c/', '/pulp/api/v3/content/rpm/advisories/6de042b9-7eb3-485c-a49b-b7511d742ec6/', '/pulp/api/v3/content/rpm/advisories/b2334511-27a5-46e3-890e-9ddca1c6a974/', '/pulp/api/v3/content/rpm/advisories/1a60b4ae-03b1-4519-b5f6-98ac1d55a844/', '/pulp/api/v3/content/rpm/advisories/8b5196fe-2846-4250-9bec-449306fd3efd/', '/pulp/api/v3/content/rpm/advisories/a9449704-2ee1-4d81-9131-791c6abda6c7/', '/pulp/api/v3/content/rpm/advisories/fb3d2f44-488c-4b2c-8372-808951d51030/', '/pulp/api/v3/content/rpm/advisories/bb8f04ae-b5e9-4a2a-83a8-e72c9dfe523f/', '/pulp/api/v3/content/rpm/advisories/e64024f6-188e-46b9-85ab-c52ece9ed146/', '/pulp/api/v3/content/rpm/advisories/79b0cb71-bad1-42d9-9f2e-d2bb4370a193/', '/pulp/api/v3/content/rpm/advisories/418fcc4c-4cba-4962-84e9-beade6d87297/', '/pulp/api/v3/content/rpm/advisories/3a7c9c38-26e5-401b-a770-eebe9348bd47/', '/pulp/api/v3/content/rpm/advisories/532630a0-8511-4a82-8ae9-b03ccd891b7a/', '/pulp/api/v3/content/rpm/advisories/5ab1679c-e4a7-46a4-9869-15a4230a09f0/', '/pulp/api/v3/content/rpm/advisories/239a2bff-6c62-4a2c-b306-84322d9c0060/', '/pulp/api/v3/content/rpm/advisories/a3b907b1-3446-4a46-9399-2e6e02d65b9d/', '/pulp/api/v3/content/rpm/advisories/0dc867a9-60f6-4f7e-b928-bc5171776120/', '/pulp/api/v3/content/rpm/advisories/6a447d71-4b80-4faa-b614-3cb6ba79a114/', '/pulp/api/v3/content/rpm/advisories/56a80ec2-d7f9-458c-9db2-110bee0d8c96/']"]
I have tried to look up several of those id’s, and I don’t see results for any I’ve checked so far:
# foreman-rake console
Loading production environment (Rails 6.0.3.7)
irb(main):001:0> ::Katello::Repository.where(id: ::Katello::RepositoryErratum.where(erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/")).map{|r| r.root.name}
=> []
irb(main):002:0> ::Katello::Repository.where(id: ::Katello::RepositoryErratum.where(erratum_pulp3_hr
ef: "/pulp/api/v3/content/rpm/advisories/633f337e-a0f0-4c6b-ad0d-b6fdbfe470f7/")).map{|r| r.root.nam
e}
=> []
irb(main):003:0> ::Katello::Repository.where(id: ::Katello::RepositoryErratum.where(erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/11610a41-26f6-4de7-b006-59f931ebc004/")).map{|r| r.root.name}
=> []
irb(main):004:0> ::Katello::Repository.where(id: ::Katello::RepositoryErratum.where(erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/633f337e-a0f0-4c6b-ad0d-b6fdbfe470f7/")).map{|r| r.root.name}
=> []
I’ll try running a complete sync on that repository again…and try to promote it.
I also have another copy of this repo in another org, which might have something to do with this. I’ll try refreshing those too.
After full-syncing both copies of the repo in both orgs, I still seem to be getting the same errors.
@mhjacks try this instead:
::Katello::Repository.where(id: ::Katello::RepositoryErratum.where(erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/").pluck(:repository_id)).map{|r| r.root.name}
Assuming that now returns something, are the repositories reported in the broken content view? They don’t all have to be, but some of them should.
As long as you’re not surprised by the results above, I think we can continue with manual cleanup. It should be easy granted that you have the bad errata list from the error:
- Copy the list of missing content units from the error
- Run the following in the console using the copied list:
::Katello::RepositoryErratum.where(erratum_pulp3_href: <copy list of missing content units>).delete_all
for example,
::Katello::RepositoryErratum.where(erratum_pulp3_href: ['/pulp/api/v3/content/rpm/advisories/633f337e-a0f0-4c6b-ad0d-b6fdbfe470f7/', '/pulp/api/v3/content/rpm/advisories/4566ee58-c015-4551-823b-37069f2ebfa9/']).delete_all
Then, try publishing again.
1 Like
Thanks! The problem content view has now been published.
1 Like