@iballou Thank you for the suggestion, I tried it but the command still fails:
[root@foreman tmp]# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager handle-artifact-checksums
/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/hazmat/bindings/openssl/binding.py:173: CryptographyDeprecationWarning: OpenSSL version 1.0.2 is no longer supported by the OpenSSL project, please upgrade. The next version of cryptography will drop support for it.
warnings.warn(
System check identified some issues:
WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Restoring missing file /var/lib/pulp/media/artifact/49/d6fdc1ad0a446e0affee271f66ca847f2facb538203dab2a713474ce812618
Restoring missing file /var/lib/pulp/media/artifact/19/39ce298615e1d7cc6d04708f47684ae70a86f76a0476c4a8222c64b7efcf32
Restoring missing file /var/lib/pulp/media/artifact/a5/8acb04f1953606e348776360bbfbeed5946ca5ce7f68c8e798b26382accc59
Restoring missing file /var/lib/pulp/media/artifact/32/75f06333c46fe863019c14709670f0b44aa6154fcf5121a3ae1c7d941b0bd6
Restoring missing file /var/lib/pulp/media/artifact/38/142eb1ffcb6384f464bf3912f818a53322b1d069ce6ff7e1e979e58d025fd6
Restoring missing file /var/lib/pulp/media/artifact/44/64e75ffc120ddf191640aa46e386170782daeca850051e9f4d3b597ff33454
Restoring missing file /var/lib/pulp/media/artifact/70/65ee4e2b2aa44a3bc82db1d397f87c396f678b4a82f4eb66e5b74c64bb8c1e
Restoring missing file /var/lib/pulp/media/artifact/ef/a1d51e9e6a2477f732b1eb9dcb1b3cc2452aaa0d39f30e714d644e2f418761
Restoring missing file /var/lib/pulp/media/artifact/fe/2aac9dd519b4bb0b7568285aae81a5cf56fb08a18dc5064a59b01486ae74d9
Restoring missing file /var/lib/pulp/media/artifact/ba/5cb0dea77048bb6dbf64690bc70fef99f5a12396f5ae116c38210be010057c
Restoring missing file /var/lib/pulp/media/artifact/0e/00deb9f0d6962e5f3ad7ab30d09913acfad3e56c2e86a5f47b83b0daf97b0d
Restoring missing file /var/lib/pulp/media/artifact/8d/49ba47ccf417f343a248a5e58dd0ae85a5e6c773fcfe078c1f8118df5249c4
Restoring missing file /var/lib/pulp/media/artifact/e3/3666347d5affb3645a24e2882962cbdd6f6fb1e5ae11b9ef8b9d3facb33b78
Restoring missing file /var/lib/pulp/media/artifact/38/e18169537cfada9626d0b2d08061db4ef2f922ff0cfcc13e67de8b9ad2c863
Restoring missing file /var/lib/pulp/media/artifact/5b/3f1344d732759905e70352ee390557fcbfcd9f1f18bd4ef07de140fa5602f4
Restoring missing file /var/lib/pulp/media/artifact/5d/4c6d4b08d4248cb4aa146aa1105adcfa391b5f8c57ef765cd585112f398b8e
Restoring missing file /var/lib/pulp/media/artifact/f6/832af96423097a574b288aea0748eab92c2dcf43e63e006ace9fd331f1bed1
Restoring missing file /var/lib/pulp/media/artifact/23/cf00d60ab7e28b3bdedb95be0eea7f93dfbc0eda4aaecc3dd241eac7ee0628
Restoring missing file /var/lib/pulp/media/artifact/11/1bad70a9c4dcd5e477c992e426498a7a728245cf860fdbad464d8915e6bc73
Restoring missing file /var/lib/pulp/media/artifact/44/6209516c0f0cf62ca07a9d2b17c0c79c472363604e0f76f4fc07c51b607c7b
Restoring missing file /var/lib/pulp/media/artifact/bd/ce51a03e32c987261652b0d6a0eff94bf6be7f0c667df7e874fe952b3f31f3
Restoring missing file /var/lib/pulp/media/artifact/fe/342ab3d7e3b76f867992f503fe336d2325e614d4cc638cba79dac42425b1c9
Restoring missing file /var/lib/pulp/media/artifact/0f/274297e424c0a3f479f041c377d5ede8551d583b4cd485a4cd945cc70b5c56
Restoring missing file /var/lib/pulp/media/artifact/28/7c2b230b29604ac92dd3497d39c265c16f17472244b71d7b737ea383c0588e
Restoring missing file /var/lib/pulp/media/artifact/6e/4540fe9645f11a9e24fed4bf4ab6f49eba215a48b2dd4fa62d8024b213da71
Restoring missing file /var/lib/pulp/media/artifact/56/f30f4fa5986bc18c745e7490374ac0d8457b4d0e04c1b6b0d03c9329e6d20e
Restoring missing file /var/lib/pulp/media/artifact/5d/0a1b5ac0964ae22d9f860f9c7f0ce1933c9dbed54cdcdd5c2b83c40ec0b51d
Restoring missing file /var/lib/pulp/media/artifact/69/623f8df8899becb30009257b452856a53e00571f7eadb815eede7920fd6e14
Restoring missing file /var/lib/pulp/media/artifact/fa/0375565a86878cac461bde4be235351ffa241d7d4a6db05dc81a375f1476c6
Restoring missing file /var/lib/pulp/media/artifact/38/7c721305afad30056b43f00380eb4bf640c36197aba68f4a1128821ee7eeb5
Restoring missing file /var/lib/pulp/media/artifact/4e/6d946c89cc23375fbe38b001f0df4aa668eeaae8c633588304f807aa53f9b7
Restoring missing file /var/lib/pulp/media/artifact/d1/5dd104f59377c8df175605e39778a9441234a8237c04ad8d178493f76c0d08
Restoring missing file /var/lib/pulp/media/artifact/d2/af432843f4db506b142414f331a8cdaa905ca8b5895ec0de7b66c0c21d8bc4
Restoring missing file /var/lib/pulp/media/artifact/87/4adadc57f8e0b67efe3996ae6805257a51fca7a8e1842667af8cf27d26c998
Restoring missing file /var/lib/pulp/media/artifact/f1/318e1bfb7bf43524fafbae51811cf545f620830e05cf116bde2ebe95306656
Restoring missing file /var/lib/pulp/media/artifact/e1/c4ee8bfa90b94384db5ca0efa3ccfb933b9e332db3e720673f1c5598e6bd96
Restoring missing file /var/lib/pulp/media/artifact/ff/364c6868f892212307e6870a79d8dd5e702f5b39370695a9f7364fb16f8735
Restoring missing file /var/lib/pulp/media/artifact/2b/4d5f7245b3c09a833c5f25b38c8c5de74d124883a033fce3da012de2648957
Restoring missing file /var/lib/pulp/media/artifact/15/572c0e0f34a88a4ad9405ec5ed933ca99afc2b8452d740ba801d5caf33d9cf
Restoring missing file /var/lib/pulp/media/artifact/7b/d3ad626347f1ac109ded3dafe350cb147a5bcb6210d599177c8088f94d16c7
Restoring missing file /var/lib/pulp/media/artifact/94/ba6a1118062a00f15c8d5f86d55c6a086a4cfd52ba025d10359340e4f72041
Restoring missing file /var/lib/pulp/media/artifact/96/888a117207ec544f0312e1110ea80c864be506bc9f73bd1acb9361b622a8ac
Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64'))
pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64'))
Traceback (most recent call last):
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 181, in handle
for chunk in fp.chunks(CHUNK_SIZE):
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/base.py", line 55, in chunks
self.seek(0)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/utils.py", line 46, in <lambda>
seek = property(lambda self: self.file.seek)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 45, in _get_file
self._file = self.storage.open(self.name, 'rb')
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 38, in open
return self._open(name, mode)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 238, in _open
return File(open(self.path(name), mode))
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/96/888a117207ec544f0312e1110ea80c864be506bc9f73bd1acb9361b622a8ac'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/theforeman/tfm-pulpcore/root/usr/bin/pulpcore-manager", line 11, in <module>
load_entry_point('pulpcore==3.16.3', 'console_scripts', 'pulpcore-manager')()
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
execute_from_command_line(sys.argv)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 186, in handle
restored = self._download_artifact(a, checksum, file_path)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 127, in _download_artifact
dl_result = downloader.fetch()
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/base.py", line 180, in fetch
return done.pop().result()
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 259, in run
return await download_wrapper()
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry
ret = await target(*args, **kwargs)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 255, in download_wrapper
return await self._run(extra_data=extra_data)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
self.raise_for_status(response)
File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
response.raise_for_status()
File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/client_reqrep.py", line 998, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64')
But we get a clearer picture now. There are artifacts inside the DB, which no longer exist upstream. The repository is set to âcontent onlyâ, so it should remove them. I tried a regular sync before, but apparently this sync did not remove the no longer existing content.
According to the âRepairing Pulpâ link you sent before its clearly stated, that the repair does not deal with this kind of situation. To be honest, I donât know why, because a full sync should just remove all local artifacts and DB entries for content no longer available upstream, but since I have no deeper understanding of Pulp, I am sure there are good reasons for that.
Would it be an option to create a new repository with a different uplink mirror? I remember I tried this before (older Katello release), and it looked to me as if Pulp is too smart, not downloading again whatâs already in the DB, so I ended up in the same place.
