Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2)

Some more info:
I tried to manually remove the affected package shown above from the repo:

[root@foreman tmp]# hammer package list --search anope-mysql-2.0.9-3.el7.x86_64.rpm
------|------------------------------------|--------------------------
ID    | FILENAME                           | SOURCE RPM
------|------------------------------------|--------------------------
62691 | anope-mysql-2.0.9-3.el7.x86_64.rpm | anope-2.0.9-3.el7.src.rpm
------|------------------------------------|--------------------------
[root@foreman tmp]# hammer repository remove-content --id 2454 --ids 62691
Repository content removed.

and after ran the handle-artifact-checksum command again, hoping it would realize, that this package is no longer part of the repo. Unfortunately, it did not as it tried to download it again and failed with the same error I showed before.

Pulp folks are thinking perhaps there are unused repository versions that are holding this old content. I saw you tried orphan cleanup a while back, but have you tried running it right before running handle-artifact-checksums?

I’m going to guess you’ll run into the same issue, but I’d say it’s worth a try.

md5 issues aside, I was able to fix my machine that was missing an artifact by deleting the RPM from the Katello UI and performing a complete sync on the repository. I only saw what happens when you try “verify content checksum”. What happens when you do a complete sync? More md5 errors?

Edit: turns out I did eventually run into my own handle-artifact-checksums error with the missing artifacts:

CommandError: Some files that were missing could not be restored: {'/var/lib/pulp/media/artifact/8d/319905eedb5a5247e3a3525889a28d5d64e822007413d02587b6f51ab3b65a', '/var/lib/pulp/media/artifact/af/478132f882e42d4f55b953cb846827c48f2d29b7c6e9bba898a61ddc27b589'}

In my case these files weren’t attempted to be redownloaded.

It indeed does download those artifacts, until it hits the error. When I run the command a second time it immediately hits the error, so the downloaded files until the error seem to get corrected.

There are absolutely multiple angles here, so I will do each of the sync tasks plus the clean orphans plus the handle-artifact-checksums in order to show you the results. Hopefully that gives you a better picture.

Task 1: foreman-rake katello:delete_orphaned_content RAILS_ENV=production

Leads to an immediate error because my smart proxy has double-encrypted hashes which I need to fix first once the new Katello update has been pushed out.

Actions::Pulp3::OrphanCleanup::DeleteOrphanRemotes

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 500
Response headers: {"date"=>"Tue, 22 Mar 2022 09:05:47 GMT", "server"=>"gunicorn", "content-type"=>"text/html", "x-frame-options"=>"DENY", "content-length"=>"145", "vary"=>"Cookie", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"053a3b72f5a546f38e39f206cd03566b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 proxy.local.lan", "connection"=>"close"}
Response body: 
<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

Task 2: regular sync via GUI

Result: success

No content added.
Total steps: 18781/18781
--------------------------------
Associating Content: 80/80
Downloading Artifacts: 67/67
Downloading Metadata Files: 6/6
Parsed Advisories: 4757/4757
Parsed Comps: 41/41
Parsed Packages: 13747/13747
Un-Associating Content: 83/83

Task 3: handle-artifact-checksums

Result: downloaded a few more and bailed out at the same missing package as before:

[root@foreman tmp]# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager handle-artifact-checksums
/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/hazmat/bindings/openssl/binding.py:173: CryptographyDeprecationWarning: OpenSSL version 1.0.2 is no longer supported by the OpenSSL project, please upgrade. The next version of cryptography will drop support for it.
  warnings.warn(
System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Restoring missing file /var/lib/pulp/media/artifact/0f/1d5193d235f0806d8d1595d0937fff20ae9aadf8fc79d2545572889893c460
Restoring missing file /var/lib/pulp/media/artifact/1f/e1fbce5e7155647da147011edef091cbb374c32efce13b6f28ba6bf2a181aa
Restoring missing file /var/lib/pulp/media/artifact/2b/4f5157446aa2ae105239598d9eb04ea665453bf99da410682878d74a2fa144
Restoring missing file /var/lib/pulp/media/artifact/67/d4dcbe5833a282e89cfdaa4d193e4c936964ccf0eae8563e04f4e14b441760
Restoring missing file /var/lib/pulp/media/artifact/a0/151298f2a668e776eee1b9bb71e14e77822ecbdf9ea75fe0f2b8d64c0bc6d2
Restoring missing file /var/lib/pulp/media/artifact/c7/0a5e4d6a5d61657788249f048846456aea304d41e174ad5c806aaaafe58e24
Restoring missing file /var/lib/pulp/media/artifact/c7/9c01bd945e8647f16786cfb7e1e4cab1e5dc6fbcc17b1fd67004eec1406799
Restoring missing file /var/lib/pulp/media/artifact/d9/271cfe2212e39eedc894813510053da9be793019b40ed66122e40a8f8e908e
Restoring missing file /var/lib/pulp/media/artifact/e8/59052c1ff0343981f9da788915f37170d2c5e9203c8f7316a18d3c8f9c5621
Restoring missing file /var/lib/pulp/media/artifact/96/b536329190da41fb6479a43a1f4f474699657e9c5243f5326608727db1ddc0
Restoring missing file /var/lib/pulp/media/artifact/a0/30b8804baa2dfdcf2305bd17a41111eb4e7200617fdf600669ac5ac6824f4b
Restoring missing file /var/lib/pulp/media/artifact/bc/5a4b2ce186c5a7dc8167bed7d757e7813e603b2a42988dd9cd8796d0f77e32
Restoring missing file /var/lib/pulp/media/artifact/2e/2b6bdd00b5b89a3fa0e54fe28bb2726d02a912f3cf790273f1cfdf12cf3447
Restoring missing file /var/lib/pulp/media/artifact/81/e751cfac5fd31a393e4049d9d7d9b90bb5a5fda052905281a400ed33c5ffc6
Restoring missing file /var/lib/pulp/media/artifact/96/5a232e63098009b412e7c2199d5f2f381826531ceee0fc8959aa3f1d799b28
Restoring missing file /var/lib/pulp/media/artifact/4b/21a74d19f6a428a5b8639ac8ed2617e83eb7a58d5f90f8b6292216245eeacb
Restoring missing file /var/lib/pulp/media/artifact/96/888a117207ec544f0312e1110ea80c864be506bc9f73bd1acb9361b622a8ac
Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64'))
pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64'))
Traceback (most recent call last):
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 181, in handle
    for chunk in fp.chunks(CHUNK_SIZE):
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/base.py", line 55, in chunks
    self.seek(0)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/utils.py", line 46, in <lambda>
    seek = property(lambda self: self.file.seek)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 45, in _get_file
    self._file = self.storage.open(self.name, 'rb')
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 238, in _open
    return File(open(self.path(name), mode))
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/96/888a117207ec544f0312e1110ea80c864be506bc9f73bd1acb9361b622a8ac'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/theforeman/tfm-pulpcore/root/usr/bin/pulpcore-manager", line 11, in <module>
    load_entry_point('pulpcore==3.16.3', 'console_scripts', 'pulpcore-manager')()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 186, in handle
    restored = self._download_artifact(a, checksum, file_path)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 127, in _download_artifact
    dl_result = downloader.fetch()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/base.py", line 180, in fetch
    return done.pop().result()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 259, in run
    return await download_wrapper()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 255, in download_wrapper
    return await self._run(extra_data=extra_data)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
    self.raise_for_status(response)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
    response.raise_for_status()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/client_reqrep.py", line 998, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64')

Task 4: Synchronize: Skip Metadata Check

Result: success

No content added.
Total steps: 18551/18551
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading Metadata Files: 6/6
Parsed Advisories: 4757/4757
Parsed Comps: 41/41
Parsed Packages: 13747/13747
Un-Associating Content: 0/0

Task 5: handle-artifacts-checksums

Result: failed

[root@foreman tmp]# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager handle-artifact-checksums
/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/hazmat/bindings/openssl/binding.py:173: CryptographyDeprecationWarning: OpenSSL version 1.0.2 is no longer supported by the OpenSSL project, please upgrade. The next version of cryptography will drop support for it.
  warnings.warn(
System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Restoring missing file /var/lib/pulp/media/artifact/ec/6c0967edd972042984ebfc1e4ee77fd1292b08e00539700edcf0f3a3bb44a6
Restoring missing file /var/lib/pulp/media/artifact/96/888a117207ec544f0312e1110ea80c864be506bc9f73bd1acb9361b622a8ac
Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64'))
pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64'))
Traceback (most recent call last):
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 181, in handle
    for chunk in fp.chunks(CHUNK_SIZE):
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/base.py", line 55, in chunks
    self.seek(0)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/utils.py", line 46, in <lambda>
    seek = property(lambda self: self.file.seek)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 45, in _get_file
    self._file = self.storage.open(self.name, 'rb')
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 238, in _open
    return File(open(self.path(name), mode))
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/96/888a117207ec544f0312e1110ea80c864be506bc9f73bd1acb9361b622a8ac'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/theforeman/tfm-pulpcore/root/usr/bin/pulpcore-manager", line 11, in <module>
    load_entry_point('pulpcore==3.16.3', 'console_scripts', 'pulpcore-manager')()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 186, in handle
    restored = self._download_artifact(a, checksum, file_path)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 127, in _download_artifact
    dl_result = downloader.fetch()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/base.py", line 180, in fetch
    return done.pop().result()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 259, in run
    return await download_wrapper()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 255, in download_wrapper
    return await self._run(extra_data=extra_data)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
    self.raise_for_status(response)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
    response.raise_for_status()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/client_reqrep.py", line 998, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://ftp.halifax.rwth-aachen.de/fedora-epel/7/x86_64/Packages/a/anope-mysql-2.0.9-3.el7.x86_64.rpm?repo=epel-7&arch=x86_64')

Task 6: Verify checksum via GUI

Result: warning

Total steps: 6/6
--------------------------------
Identify corrupted units: 0/0
Identify missing units: 6/6
Repair corrupted units: 0/0
Katello::Errors::Pulp3Error: 'md5'

@iballou that’s all I can try from my end…

Thanks for all the info @rbremer. So I think the next best step would be to get Orphan Cleanup working. I see your task is failing at the DeleteOrphanRemotes step. Is the Actions::Pulp3::OrphanCleanup::DeleteOrphanRepositoryVersions action running to completion for the “Primary” smart proxy (the one likely running on your Katello server)?

If you’re talking about this encryption issue (PulpRpmClient::ApiError HTTP 500 during sync repository - #32 by iballou), then you should be able to run the repair script soon. The latest Pulpcore update got merged into our packaging repo yesterday.

Thank you Ian, I will wait for them to appear and clean this up first. Afterwards I will resume with my tests here and let you know.

1 Like

@iballou I could update to the new pulp RPMs yesterday:

===========================================================================================================================================================================================================================================================================================================================================================
 Package                                                                                              Arch                                                                     Version                                                                                    Repository                                                                  Size
===========================================================================================================================================================================================================================================================================================================================================================
Updating:
 httpd                                                                                                x86_64                                                                   2.4.6-97.el7.centos.5                                                                      updates                                                                    2.7 M
 httpd-tools                                                                                          x86_64                                                                   2.4.6-97.el7.centos.5                                                                      updates                                                                     94 k
 mod_ssl                                                                                              x86_64                                                                   1:2.4.6-97.el7.centos.5                                                                    updates                                                                    115 k
 tfm-pulpcore-python3-aiohttp                                                                         x86_64                                                                   3.8.1-2.el7                                                                                pulpcore                                                                   608 k
 tfm-pulpcore-python3-async-timeout                                                                   noarch                                                                   4.0.2-1.el7                                                                                pulpcore                                                                    13 k
 tfm-pulpcore-python3-pulpcore                                                                        noarch                                                                   3.16.6-1.el7                                                                               pulpcore                                                                   687 k
 tzdata                                                                                               noarch                                                                   2022a-1.el7                                                                                updates                                                                    501 k
 tzdata-java                                                                                          noarch                                                                   2022a-1.el7                                                                                updates                                                                    190 k
Installing for dependencies:
 tfm-pulpcore-python3-aiosignal                                                                       noarch                                                                   1.2.0-1.el7                                                                                pulpcore                                                                    14 k
 tfm-pulpcore-python3-brotli                                                                          x86_64                                                                   1.0.9-1.el7                                                                                pulpcore                                                                   305 k
 tfm-pulpcore-python3-cchardet                                                                        x86_64                                                                   2.1.7-1.el7                                                                                pulpcore                                                                    99 k
 tfm-pulpcore-python3-frozenlist                                                                      x86_64                                                                   1.3.0-1.el7                                                                                pulpcore                                                                    40 k

Transaction Summary
===========================================================================================================================================================================================================================================================================================================================================================
Install             ( 4 Dependent packages)
Upgrade  8 Packages

Did this on both the foreman and the proxy server. After I restarted both and ran the command

sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager datarepair-2327 --dry-run

on the proxy. It showed - as expected - a lot of double encrypted entries and without the --dry-run fixed them.
Running it on the foreman server showed 0 issues, also as expected, as I am below 100 entries in that table.

After I tried the remove_orphan command again, and it ran successfully for the smart proxy, however, the second task (apparently running on the foreman server) did run into a timeout error. So I dug a little deeper and it looks like the new RPMs are causing a startup error for gunicorn related to the ‘md5’ issue I have. This is what happened after rebooting the foreman server in /v/l/messages:

Mar 27 11:18:13 foreman pulpcore-content: raise ImproperlyConfigured(
Mar 27 11:18:13 foreman pulpcore-content: django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'md5'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31414] [INFO] Worker exiting (pid: 31414)
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31413] [ERROR] Exception in worker process
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Mar 27 11:18:13 foreman pulpcore-content: worker.init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/worker.py", line 51, in init_process
Mar 27 11:18:13 foreman pulpcore-content: super().init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Mar 27 11:18:13 foreman pulpcore-content: self.load_wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.wsgi = self.app.wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.callable = self.load()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Mar 27 11:18:13 foreman pulpcore-content: return self.load_wsgiapp()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Mar 27 11:18:13 foreman pulpcore-content: return util.import_app(self.app_uri)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/__init__.py", line 19, in <module>
Mar 27 11:18:13 foreman pulpcore-content: django.setup()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/__init__.py", line 19, in setup
Mar 27 11:18:13 foreman pulpcore-content: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
Mar 27 11:18:13 foreman pulpcore-content: self._setup(name)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
Mar 27 11:18:13 foreman pulpcore-content: self._wrapped = Settings(settings_module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(self.SETTINGS_MODULE)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 415, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise e
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 374, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise ImproperlyConfigured(
Mar 27 11:18:13 foreman pulpcore-content: django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'md5'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31413] [INFO] Worker exiting (pid: 31413)
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 209, in run
Mar 27 11:18:13 foreman pulpcore-content: self.sleep()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 357, in sleep
Mar 27 11:18:13 foreman pulpcore-content: ready = select.select([self.PIPE[0]], [], [], 1.0)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Mar 27 11:18:13 foreman pulpcore-content: self.reap_workers()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Mar 27 11:18:13 foreman pulpcore-content: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Mar 27 11:18:13 foreman pulpcore-content: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Mar 27 11:18:13 foreman pulpcore-content: During handling of the above exception, another exception occurred:
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/bin/gunicorn", line 11, in <module>
Mar 27 11:18:13 foreman pulpcore-content: load_entry_point('gunicorn==20.1.0', 'console_scripts', 'gunicorn')()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
Mar 27 11:18:13 foreman pulpcore-content: WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
Mar 27 11:18:13 foreman pulpcore-content: super().run()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Mar 27 11:18:13 foreman pulpcore-content: Arbiter(self).run()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 229, in run
Mar 27 11:18:13 foreman pulpcore-content: self.halt(reason=inst.reason, exit_status=inst.exit_status)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 342, in halt
Mar 27 11:18:13 foreman pulpcore-content: self.stop()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 393, in stop
Mar 27 11:18:13 foreman pulpcore-content: time.sleep(0.1)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Mar 27 11:18:13 foreman pulpcore-content: self.reap_workers()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Mar 27 11:18:13 foreman pulpcore-content: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Mar 27 11:18:13 foreman pulpcore-content: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Mar 27 11:18:13 foreman systemd: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
Mar 27 11:18:13 foreman systemd: Unit pulpcore-worker@3.service entered failed state.
Mar 27 11:18:13 foreman systemd: pulpcore-worker@3.service failed.
Mar 27 11:18:13 foreman systemd: pulpcore-content.service: main process exited, code=exited, status=1/FAILURE
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31418] [ERROR] Exception in worker process
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Mar 27 11:18:13 foreman pulpcore-content: worker.init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/worker.py", line 51, in init_process
Mar 27 11:18:13 foreman pulpcore-content: super().init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Mar 27 11:18:13 foreman pulpcore-content: self.load_wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.wsgi = self.app.wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.callable = self.load()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Mar 27 11:18:13 foreman pulpcore-content: return self.load_wsgiapp()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Mar 27 11:18:13 foreman pulpcore-content: return util.import_app(self.app_uri)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/__init__.py", line 19, in <module>
Mar 27 11:18:13 foreman pulpcore-content: django.setup()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/__init__.py", line 19, in setup
Mar 27 11:18:13 foreman pulpcore-content: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
Mar 27 11:18:13 foreman pulpcore-content: self._setup(name)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
Mar 27 11:18:13 foreman pulpcore-content: self._wrapped = Settings(settings_module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(self.SETTINGS_MODULE)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 415, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise e
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 374, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise ImproperlyConfigured(
Mar 27 11:18:13 foreman pulpcore-content: django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'md5'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31418] [INFO] Worker exiting (pid: 31418)
Mar 27 11:18:13 foreman systemd: Unit pulpcore-content.service entered failed state.
Mar 27 11:18:13 foreman systemd: pulpcore-content.service failed.
Mar 27 11:18:14 foreman systemd: pulpcore-api.service holdoff time over, scheduling restart.
Mar 27 11:18:14 foreman systemd: Stopped Pulp API Server.

I tried the handle-artifact-checksums command again but it still fails at the same file missing from the upstream repo. It surely looks like we are getting closer and closer, but without being able to fully remove the outdated RPMs from the database I am still stuck.

Getting the pulp API server back online is probably the most important step now and I hope you have an idea about that. Thank you.

I was looking at the line raising the ‘md5’ checksum missing exception, it looks like it checks all artifacts for any checksum being null, so I tried to run that command for all checksums, this is the result:

[root@foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE md5 IS NULL;'
 count
-------
 50613
(1 row)

[root@ foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE md5 IS NOT NULL;'
 count
-------
  1982
(1 row)

[root@ foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE sha1 IS NULL;'
 count
-------
     0
(1 row)

[root@ foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE sha1 IS NOT NULL;'
 count
-------
 52595
(1 row)

[root@ foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE sha256 IS NOT NULL;'
 count
-------
 52595
(1 row)

[root@ foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE sha256 IS NULL;'
 count
-------
     0
(1 row)

[root@ foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE sha512 IS NULL;'
 count
-------
     0
(1 row)

[root@ foreman tmp]# sudo -u postgres psql pulpcore -c 'SELECT count(pulp_id) FROM core_artifact WHERE sha512 IS NOT NULL;'
 count
-------
 52595
(1 row)

So somehow MD5 seems to be in the list of enabled checksums, as this line here is iterating over all configured checksums:
file: /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py

    369             for checksum in ALLOWED_CONTENT_CHECKSUMS:
    370                 # can't import Artifact here so use a direct db connection
    371                 cursor.execute(f"SELECT count(pulp_id) FROM core_artifact WHERE {checksum} IS NULL")
    372                 row = cursor.fetchone()
    373                 if row[0] > 0:
    374                     raise ImproperlyConfigured(
    375                         _(
    376                             "There have been identified artifacts missing checksum '{}'. "
    377                             "Run 'pulpcore-manager handle-artifact-checksums' first to populate "
    378                             "missing artifact checksums."
    379                         ).format(checksum)
    380                     )

Maybe it was disabled in an earlier release of Katello/Foreman (please remember, my system is pretty old and has been upgraded a lot), and later got enabled? Can I update the column and set it to NULL and remove MD5 from the list of allowed checksums manually? As I am not sure the foreman-installer command to set the enabled checksums will run at that point in time, as I still have this orphaned files in the DB. What do you think?

Hey @rbremer,

The installer flag to enable it is the following, which was in the foreman-installer command from earlier:

--foreman-proxy-content-pulpcore-allowed-content-checksums md5

I suggested running that before because it looked like you had MD5 content synced which was causing errors. Now it might be best to try disabling it, restarting pulpcore* --all services, and re-running orphan cleanup.

To get the checksums back to the default (no MD5), run the following:

foreman-installer --reset-foreman-proxy-content-pulpcore-allowed-content-checksums

I’ll also mention that the checksum settings live in /etc/pulp/settings.py

@iballou thank you again for your help.

I finally managed to get it all back in order, this is what I did:

  1. update to the latest RPMs (which included the pulp core fix for double encrypted entries)
  2. rebooted foreman and the proxy
  3. ran the datarepair-2327 on the proxy, it repaired some entries
  4. ran it again on the proxy, this time 0 errors
  5. ran remove-orphan, which succeeded for the proxy but timed out on the foreman server
  6. discovered, that pulp-core constantly restarts on the foreman server and identified the cause (md5 is enabled and 40k+ entries didn’t have md5 hashes set)
  7. did something nasty, ran SQL to set the md5 has to the value “1” for all NULL (I won’t post the SQL command here as I don’t wanna encourage anyone being as “stupidly brave”
  8. pulp-core did now restart fine
  9. tried the remove-orphan again, this time it worked on both foreman and the proxy
  10. handle-artifacts-checksum still failed with that one file which was removed from upstream
  11. did a repository sync on ALL my repositories (normal sync), as they are set to “data only” for mirroring, removed upstream content is also removed
  12. discovered, that the RPM in question is still visible in the Foreman GUI
  13. published and promoted new version of each and every content view
  14. deleted all previous version of each and every content view
  15. ran remove-orphan again, successful on both the proxy and foreman server
  16. used SQL (again boldly brave) to reset the MD5 hash to NULL where it was “1”
  17. ran handle-artifact-checksums and for the first time it completed successfully, regenerating all MD5 checksums and downloading thousands of missing files (those didn’t survive the pulp2 → pulp3 migration)
  18. ran SQL to check if I have missing MD5 checksums: 0!
  19. full sync of the proxy (still running)

So tomorrow I will check for the result of 19 and if all is fine, will be finally back with a fully functioning Foreman/Katello system.

Big thanks to everyone, who helped me throughout this process. Hopefully nobody else encounters those issues after updating too quickly from 3 to 4.

3 Likes

@rbremer I’m glad to hear that you got handle-artifact-checksums working and it seems to have corrected those long-missing artifacts. I’ll be curious to hear about your proxy sync, that’ll be a good test of whether or not your content is all set.

These steps you ran through will be great to have in the record books. @dralley you might be interested to see the above since some Pulpcore DB tinkering was involved. It would be nice to know why the DB modification was necessary and if any future users with the same issue can avoid it.

2 Likes

Proxy sync has finished without errors, so I am happy to report that all looks good now. I will disable MD5 checksums through the foreman-installer, having never turned it on deliberately, so maybe it was enabled by default way back in time.

All I find inside of core_artifacts now are 27 entries pointing to “repomd.xml”, created and last updated back in 3/2020, which can’t be recovered by handle-artifact-checksums. Gonna ignore these for now and hopefully pulp will never touch them.

1 Like