@iballou I could update to the new pulp RPMs yesterday:
===========================================================================================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================================================================================================================================================================================================================================================
Updating:
httpd x86_64 2.4.6-97.el7.centos.5 updates 2.7 M
httpd-tools x86_64 2.4.6-97.el7.centos.5 updates 94 k
mod_ssl x86_64 1:2.4.6-97.el7.centos.5 updates 115 k
tfm-pulpcore-python3-aiohttp x86_64 3.8.1-2.el7 pulpcore 608 k
tfm-pulpcore-python3-async-timeout noarch 4.0.2-1.el7 pulpcore 13 k
tfm-pulpcore-python3-pulpcore noarch 3.16.6-1.el7 pulpcore 687 k
tzdata noarch 2022a-1.el7 updates 501 k
tzdata-java noarch 2022a-1.el7 updates 190 k
Installing for dependencies:
tfm-pulpcore-python3-aiosignal noarch 1.2.0-1.el7 pulpcore 14 k
tfm-pulpcore-python3-brotli x86_64 1.0.9-1.el7 pulpcore 305 k
tfm-pulpcore-python3-cchardet x86_64 2.1.7-1.el7 pulpcore 99 k
tfm-pulpcore-python3-frozenlist x86_64 1.3.0-1.el7 pulpcore 40 k
Transaction Summary
===========================================================================================================================================================================================================================================================================================================================================================
Install ( 4 Dependent packages)
Upgrade 8 Packages
Did this on both the foreman and the proxy server. After I restarted both and ran the command
sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager datarepair-2327 --dry-run
on the proxy. It showed - as expected - a lot of double encrypted entries and without the --dry-run fixed them.
Running it on the foreman server showed 0 issues, also as expected, as I am below 100 entries in that table.
After I tried the remove_orphan command again, and it ran successfully for the smart proxy, however, the second task (apparently running on the foreman server) did run into a timeout error. So I dug a little deeper and it looks like the new RPMs are causing a startup error for gunicorn related to the ‘md5’ issue I have. This is what happened after rebooting the foreman server in /v/l/messages:
Mar 27 11:18:13 foreman pulpcore-content: raise ImproperlyConfigured(
Mar 27 11:18:13 foreman pulpcore-content: django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'md5'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31414] [INFO] Worker exiting (pid: 31414)
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31413] [ERROR] Exception in worker process
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Mar 27 11:18:13 foreman pulpcore-content: worker.init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/worker.py", line 51, in init_process
Mar 27 11:18:13 foreman pulpcore-content: super().init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Mar 27 11:18:13 foreman pulpcore-content: self.load_wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.wsgi = self.app.wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.callable = self.load()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Mar 27 11:18:13 foreman pulpcore-content: return self.load_wsgiapp()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Mar 27 11:18:13 foreman pulpcore-content: return util.import_app(self.app_uri)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/__init__.py", line 19, in <module>
Mar 27 11:18:13 foreman pulpcore-content: django.setup()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/__init__.py", line 19, in setup
Mar 27 11:18:13 foreman pulpcore-content: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
Mar 27 11:18:13 foreman pulpcore-content: self._setup(name)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
Mar 27 11:18:13 foreman pulpcore-content: self._wrapped = Settings(settings_module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(self.SETTINGS_MODULE)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 415, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise e
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 374, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise ImproperlyConfigured(
Mar 27 11:18:13 foreman pulpcore-content: django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'md5'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31413] [INFO] Worker exiting (pid: 31413)
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 209, in run
Mar 27 11:18:13 foreman pulpcore-content: self.sleep()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 357, in sleep
Mar 27 11:18:13 foreman pulpcore-content: ready = select.select([self.PIPE[0]], [], [], 1.0)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Mar 27 11:18:13 foreman pulpcore-content: self.reap_workers()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Mar 27 11:18:13 foreman pulpcore-content: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Mar 27 11:18:13 foreman pulpcore-content: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Mar 27 11:18:13 foreman pulpcore-content: During handling of the above exception, another exception occurred:
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/bin/gunicorn", line 11, in <module>
Mar 27 11:18:13 foreman pulpcore-content: load_entry_point('gunicorn==20.1.0', 'console_scripts', 'gunicorn')()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
Mar 27 11:18:13 foreman pulpcore-content: WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
Mar 27 11:18:13 foreman pulpcore-content: super().run()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Mar 27 11:18:13 foreman pulpcore-content: Arbiter(self).run()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 229, in run
Mar 27 11:18:13 foreman pulpcore-content: self.halt(reason=inst.reason, exit_status=inst.exit_status)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 342, in halt
Mar 27 11:18:13 foreman pulpcore-content: self.stop()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 393, in stop
Mar 27 11:18:13 foreman pulpcore-content: time.sleep(0.1)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Mar 27 11:18:13 foreman pulpcore-content: self.reap_workers()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Mar 27 11:18:13 foreman pulpcore-content: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Mar 27 11:18:13 foreman pulpcore-content: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Mar 27 11:18:13 foreman systemd: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
Mar 27 11:18:13 foreman systemd: Unit pulpcore-worker@3.service entered failed state.
Mar 27 11:18:13 foreman systemd: pulpcore-worker@3.service failed.
Mar 27 11:18:13 foreman systemd: pulpcore-content.service: main process exited, code=exited, status=1/FAILURE
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31418] [ERROR] Exception in worker process
Mar 27 11:18:13 foreman pulpcore-content: Traceback (most recent call last):
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Mar 27 11:18:13 foreman pulpcore-content: worker.init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/worker.py", line 51, in init_process
Mar 27 11:18:13 foreman pulpcore-content: super().init_process()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Mar 27 11:18:13 foreman pulpcore-content: self.load_wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.wsgi = self.app.wsgi()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Mar 27 11:18:13 foreman pulpcore-content: self.callable = self.load()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Mar 27 11:18:13 foreman pulpcore-content: return self.load_wsgiapp()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Mar 27 11:18:13 foreman pulpcore-content: return util.import_app(self.app_uri)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/__init__.py", line 19, in <module>
Mar 27 11:18:13 foreman pulpcore-content: django.setup()
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/__init__.py", line 19, in setup
Mar 27 11:18:13 foreman pulpcore-content: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
Mar 27 11:18:13 foreman pulpcore-content: self._setup(name)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
Mar 27 11:18:13 foreman pulpcore-content: self._wrapped = Settings(settings_module)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
Mar 27 11:18:13 foreman pulpcore-content: mod = importlib.import_module(self.SETTINGS_MODULE)
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
Mar 27 11:18:13 foreman pulpcore-content: return _bootstrap._gcd_import(name[level:], package, level)
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
Mar 27 11:18:13 foreman pulpcore-content: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 415, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise e
Mar 27 11:18:13 foreman pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 374, in <module>
Mar 27 11:18:13 foreman pulpcore-content: raise ImproperlyConfigured(
Mar 27 11:18:13 foreman pulpcore-content: django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'md5'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Mar 27 11:18:13 foreman pulpcore-content: [2022-03-27 09:18:13 +0000] [31418] [INFO] Worker exiting (pid: 31418)
Mar 27 11:18:13 foreman systemd: Unit pulpcore-content.service entered failed state.
Mar 27 11:18:13 foreman systemd: pulpcore-content.service failed.
Mar 27 11:18:14 foreman systemd: pulpcore-api.service holdoff time over, scheduling restart.
Mar 27 11:18:14 foreman systemd: Stopped Pulp API Server.
I tried the handle-artifact-checksums command again but it still fails at the same file missing from the upstream repo. It surely looks like we are getting closer and closer, but without being able to fully remove the outdated RPMs from the database I am still stuck.
Getting the pulp API server back online is probably the most important step now and I hope you have an idea about that. Thank you.