Pulp services fail to start

Support
1

Problem:
All pulpcore services are stuck at ‘activating’ and I cannot add/modify/remove any repositories.
Expected outcome:
The pulpcore services should start and I should be able to manipulate all repositories.
Foreman and Proxy versions:
foreman-release-2.4.0-1 foreman-proxy-2.4.0-1 foreman-installer-katello-2.4.0-1 hostname -f-foreman-proxy-client-1.0-1
Foreman and Proxy plugin versions:
unknown - I do not know how this differs from previous question
Distribution and version:
CentOS Linux release 7.9.2009
Other relevant data:
I am very new to foreman and trying to build a replacement for an older version. I had a fully functional foreman+katello installation and I believe I corrupted it by running the installer on the main (new) foreman server with ‘–foreman-proxy-content-pulpcore-mirror true’. It has not been able to sync repos since then. I reran the installer with ‘–foreman-proxy-content-pulpcore-mirror false’ but the installer now fails at Upgrade Step 1/3: katello:correct_repositories…Failed upgrade task: katello:correct_repositories.

An abbriated snippet from the /var/log/messages pulpcore-content shows…
May 26 11:38:02 puppet systemd: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
May 26 11:38:02 puppet systemd: Unit pulpcore-worker@2.service entered failed state.
May 26 11:38:02 puppet systemd: pulpcore-worker@2.service failed.
May 26 11:38:02 puppet systemd: pulpcore-worker@1.service: main process exited, code=exited, status=1/FAILURE
May 26 11:38:02 puppet systemd: Unit pulpcore-worker@1.service entered failed state.
May 26 11:38:02 puppet systemd: pulpcore-worker@1.service failed.
May 26 11:38:02 puppet systemd: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
May 26 11:38:02 puppet systemd: Unit pulpcore-worker@3.service entered failed state.
May 26 11:38:02 puppet systemd: pulpcore-worker@3.service failed.
May 26 11:38:02 puppet pulpcore-resource-manager: Traceback (most recent call last):
May 26 11:38:02 puppet pulpcore-resource-manager: File “/usr/bin/rq”, line 11, in
…
May 26 11:38:02 puppet pulpcore-resource-manager: File “”, line 994, in _gcd_import
May 26 11:38:02 puppet pulpcore-resource-manager: File “”, line 971, in _find_and_load
May 26 11:38:02 puppet pulpcore-resource-manager: File “”, line 955, in _find_and_load_unlocked
May 26 11:38:02 puppet pulpcore-resource-manager: File “”, line 665, in _load_unlocked
May 26 11:38:02 puppet pulpcore-resource-manager: File “”, line 678, in exec_module
May 26 11:38:02 puppet pulpcore-resource-manager: File “”, line 219, in _call_with_frames_removed
May 26 11:38:02 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py”, line 13, in
May 26 11:38:02 puppet pulpcore-resource-manager: django.setup()
…
The /var/log/messages file is filling with pulpcore- related errors.

I haved disabled the deb content proxy (–foreman-proxy-content-enable-deb false --katello-enable-deb false) since I am not using it and do not have any deb repos now…just chasing any possible solutions I could find on the internet.

Something I notice in the repositories (looking in the foreman web UI) is the ‘Published At:’ setting is wrong, it is ‘https://localhost’…and should be the ‘hostname -f’. I can’t find a way to manually change this but suspect it is related to the overall issue.

ANY suggestions about how I can correct this is most appreciated.

2

The ‘localhost’ issue showing up in the UI is a known bug and is being worked on here: Bug #32362: katello pulp url setting in katello.yaml is no longer set by the installer and the default uses localhost rather than FQDN - Katello - Foreman

You say: The /var/log/messages file is filling with pulpcore- related errors. Are there any other relevant tracebacks? The one that you posted for pulpcore-resource-manager looks quite small (i’m wondering if there is more to it).

You might also run
journalctl -u pulpcore-worker@1

to help focus down to a single service.

2 Likes
3

Thank you for your response, below is the full set of traceback entries for a couple of pulp processes from this morning. I have a ‘dumb user’ question as I am very new to foreman+katello. The system I am trying to fix is being prepped to become a replacement for a production install of foreman(2.0.3-1)+katello(3.15.3-1)

I have not seen any instructions explaining how to install incremental upgrades for the foreman & katello products on production systems. Are there instructions for doing this?

HERE IS /var/log/messages TRACE INFO…
May 27 07:40:22 puppet pulpcore-resource-manager: Traceback (most recent call last):
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/bin/rq”, line 11, in
May 27 07:40:22 puppet pulpcore-resource-manager: load_entry_point(‘rq==1.7.0’, ‘console_scripts’, ‘rq’)()
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/click/core.py”, line 829, in call
May 27 07:40:22 puppet pulpcore-resource-manager: return self.main(*args, **kwargs)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/click/core.py”, line 782, in main
May 27 07:40:22 puppet pulpcore-resource-manager: rv = self.invoke(ctx)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1259, in invoke
May 27 07:40:22 puppet pulpcore-resource-manager: return _process_result(sub_ctx.command.invoke(sub_ctx))
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1066, in invoke
May 27 07:40:22 puppet pulpcore-resource-manager: return ctx.invoke(self.callback, **ctx.params)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/click/core.py”, line 610, in invoke
May 27 07:40:22 puppet pulpcore-resource-manager: return callback(*args, **kwargs)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/rq/cli/cli.py”, line 77, in wrapper
May 27 07:40:22 puppet pulpcore-resource-manager: cli_config = CliConfig(**kwargs)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/rq/cli/helpers.py”, line 215, in init
May 27 07:40:22 puppet pulpcore-resource-manager: self.worker_class = import_attribute(worker_class)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/rq/utils.py”, line 129, in import_attribute
May 27 07:40:22 puppet pulpcore-resource-manager: module = importlib.import_module(module_name)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
May 27 07:40:22 puppet pulpcore-resource-manager: return _bootstrap._gcd_import(name[level:], package, level)
May 27 07:40:22 puppet pulpcore-resource-manager: File “”, line 994, in _gcd_import
May 27 07:40:22 puppet pulpcore-resource-manager: File “”, line 971, in _find_and_load
May 27 07:40:22 puppet pulpcore-resource-manager: File “”, line 955, in _find_and_load_unlocked
May 27 07:40:22 puppet pulpcore-resource-manager: File “”, line 665, in _load_unlocked
May 27 07:40:22 puppet pulpcore-resource-manager: File “”, line 678, in exec_module
May 27 07:40:22 puppet pulpcore-resource-manager: File “”, line 219, in _call_with_frames_removed
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py”, line 13, in
May 27 07:40:22 puppet pulpcore-resource-manager: django.setup()
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/django/init.py”, line 19, in setup
May 27 07:40:22 puppet pulpcore-resource-manager: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
May 27 07:40:22 puppet pulpcore-resource-manager: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 79, in getattr
…
May 27 07:40:22 puppet pulpcore-worker-2: Traceback (most recent call last):
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/bin/rq”, line 11, in
May 27 07:40:22 puppet pulpcore-worker-2: load_entry_point(‘rq==1.7.0’, ‘console_scripts’, ‘rq’)()
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/click/core.py”, line 829, in call
May 27 07:40:22 puppet pulpcore-worker-2: return self.main(*args, **kwargs)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/click/core.py”, line 782, in main
May 27 07:40:22 puppet pulpcore-worker-2: rv = self.invoke(ctx)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1259, in invoke
May 27 07:40:22 puppet pulpcore-worker-2: return _process_result(sub_ctx.command.invoke(sub_ctx))
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1066, in invoke
May 27 07:40:22 puppet pulpcore-worker-2: return ctx.invoke(self.callback, **ctx.params)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/click/core.py”, line 610, in invoke
May 27 07:40:22 puppet pulpcore-worker-2: return callback(*args, **kwargs)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/cli/cli.py”, line 77, in wrapper
May 27 07:40:22 puppet pulpcore-worker-2: cli_config = CliConfig(**kwargs)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/cli/helpers.py”, line 215, in init
May 27 07:40:22 puppet pulpcore-worker-2: self.worker_class = import_attribute(worker_class)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/utils.py”, line 129, in import_attribute
May 27 07:40:22 puppet pulpcore-worker-2: module = importlib.import_module(module_name)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
May 27 07:40:22 puppet pulpcore-worker-2: return _bootstrap._gcd_import(name[level:], package, level)
May 27 07:40:22 puppet pulpcore-worker-2: File “”, line 994, in _gcd_import
May 27 07:40:22 puppet pulpcore-worker-2: File “”, line 971, in _find_and_load
May 27 07:40:22 puppet pulpcore-worker-2: File “”, line 955, in _find_and_load_unlocked
May 27 07:40:22 puppet pulpcore-worker-2: File “”, line 665, in _load_unlocked
May 27 07:40:22 puppet pulpcore-worker-2: File “”, line 678, in exec_module
May 27 07:40:22 puppet pulpcore-worker-2: File “”, line 219, in _call_with_frames_removed
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py”, line 13, in
May 27 07:40:22 puppet pulpcore-worker-2: django.setup()
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/django/init.py”, line 19, in setup
May 27 07:40:22 puppet pulpcore-worker-2: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
May 27 07:40:22 puppet pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 79, in getattr

OUTPUT OF ‘journalctl’ FOCUS ON pulpcore-worker-2…

journalctl -u pulpcore-worker@2

– Logs begin at Wed 2021-05-26 18:11:41 EDT, end at Thu 2021-05-27 07:48:24 EDT. –
May 26 18:11:45 <fqdn_here> systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
May 26 18:11:45 <fqdn_here> systemd[1]: Stopped Pulp RQ Worker.
May 26 18:11:45 <fqdn_here> systemd[1]: Started Pulp RQ Worker.
May 26 18:11:45 <fqdn_here> systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
May 26 18:11:45 <fqdn_here> systemd[1]: Unit pulpcore-worker@2.service entered failed state.
May 26 18:11:45 <fqdn_here> systemd[1]: pulpcore-worker@2.service failed.
May 26 18:11:48 <fqdn_here> systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
May 26 18:11:48 <fqdn_here> systemd[1]: Stopped Pulp RQ Worker.
May 26 18:11:48 <fqdn_here> systemd[1]: Started Pulp RQ Worker.
May 26 18:11:49 <fqdn_here> systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
May 26 18:11:49 <fqdn_here> systemd[1]: Unit pulpcore-worker@2.service entered failed state.
May 26 18:11:49 <fqdn_here> systemd[1]: pulpcore-worker@2.service failed.
May 26 18:11:52 <fqdn_here> systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
May 26 18:11:52 <fqdn_here> systemd[1]: Stopped Pulp RQ Worker.
May 26 18:11:52 <fqdn_here> systemd[1]: Started Pulp RQ Worker.
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: Traceback (most recent call last):
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/bin/rq”, line 11, in
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: load_entry_point(‘rq==1.7.0’, ‘console_scripts’, ‘rq’)()
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 829, in call
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: return self.main(*args, **kwargs)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 782, in main
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: rv = self.invoke(ctx)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1259, in invoke
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: return _process_result(sub_ctx.command.invoke(sub_ctx))
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1066, in invoke
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: return ctx.invoke(self.callback, **ctx.params)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 610, in invoke
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: return callback(*args, **kwargs)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/rq/cli/cli.py”, line 77, in wrapper
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: cli_config = CliConfig(**kwargs)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/rq/cli/helpers.py”, line 215, in init
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: self.worker_class = import_attribute(worker_class)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/rq/utils.py”, line 129, in import_attribute
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: module = importlib.import_module(module_name)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: return _bootstrap._gcd_import(name[level:], package, level)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 994, in _gcd_import
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 971, in _find_and_load
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 955, in _find_and_load_unlocked
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 665, in _load_unlocked
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 678, in exec_module
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 219, in _call_with_frames_removed
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py”, line 13, in
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: django.setup()
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/django/init.py”, line 19, in setup
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 79, in getattr
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: self._setup(name)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 66, in _setup
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: self._wrapped = Settings(settings_module)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 157, in init
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: mod = importlib.import_module(self.SETTINGS_MODULE)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: return _bootstrap._gcd_import(name[level:], package, level)
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 994, in _gcd_import
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 971, in _find_and_load
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 955, in _find_and_load_unlocked
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 665, in _load_unlocked
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 678, in exec_module
May 26 18:11:52 <fqdn_here> pulpcore-worker-2[253568]: File “”, line 219, in _call_with_frames_removed

4

Thank you for the tip about the workaround for ‘localhost’ in the UI. I was able to resolve that issue but pulp still fails to start. I don’t know if this is helpful but below are the pulp packages installed…

rpm -qa | grep pulp

python3-pulp-container-2.2.2-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.9.1-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.6.0-1.el7.noarch
python3-pulpcore-3.9.1-2.el7.noarch
tfm-rubygem-pulp_file_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-1.el7.noarch
pulp-client-1.0-1.noarch
python3-pulp-file-1.5.0-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.2.0-1.el7.noarch
python3-pulp-certguard-1.1.0-1.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python3-pulp-deb-2.9.2-1.el7.noarch
tfm-rubygem-pulpcore_client-3.9.0-1.el7.noarch

5

I think maybe the traceback is missing the bottom part? I expected the word ‘Exception’ or ‘exception’ to be present somewhere but I don’t see it. Can you try getting the traceback once more and posting it?

6

I provided everything that is in /var/log/messages…is there a specific command I can use to pull the traceback info you are trying to see that seems to be truncated? I don’t know the foreman commands very well.

7

Here is the latest messages from /var/log/messages, I believe this has the full traceback details.
I am guessing this entry ‘raise VersionConflict(dist, req).with_context(dependent_req)’ is the key to the issue but I do not know how to resolve it.

Jun 4 10:12:59 puppet systemd: Started Pulp RQ Worker.
Jun 4 10:12:59 puppet pulpcore-api: [2021-06-04 10:12:59 -0400] [210369] [ERROR] Exception in worker process
Jun 4 10:12:59 puppet pulpcore-api: Traceback (most recent call last):
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/gunicorn/arbiter.py”, line 583, in spawn_worker
Jun 4 10:12:59 puppet pulpcore-api: worker.init_process()
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/gunicorn/workers/base.py”, line 119, in init_process
Jun 4 10:12:59 puppet pulpcore-api: self.load_wsgi()
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/gunicorn/workers/base.py”, line 144, in load_wsgi
Jun 4 10:12:59 puppet pulpcore-api: self.wsgi = self.app.wsgi()
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/gunicorn/app/base.py”, line 67, in wsgi
Jun 4 10:12:59 puppet pulpcore-api: self.callable = self.load()
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py”, line 49, in load
Jun 4 10:12:59 puppet pulpcore-api: return self.load_wsgiapp()
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py”, line 39, in load_wsgiapp
Jun 4 10:12:59 puppet pulpcore-api: return util.import_app(self.app_uri)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/gunicorn/util.py”, line 358, in import_app
Jun 4 10:12:59 puppet pulpcore-api: mod = importlib.import_module(module)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
Jun 4 10:12:59 puppet pulpcore-api: return _bootstrap._gcd_import(name[level:], package, level)
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 994, in _gcd_import
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 971, in _find_and_load
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 955, in _find_and_load_unlocked
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 665, in _load_unlocked
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 678, in exec_module
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 219, in _call_with_frames_removed
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/pulpcore/app/wsgi.py”, line 17, in
Jun 4 10:12:59 puppet pulpcore-api: application = get_wsgi_application()
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/django/core/wsgi.py”, line 12, in get_wsgi_application
Jun 4 10:12:59 puppet pulpcore-api: django.setup(set_prefix=False)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/django/init.py”, line 19, in setup
Jun 4 10:12:59 puppet pulpcore-api: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 79, in getattr
Jun 4 10:12:59 puppet pulpcore-api: self._setup(name)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 66, in _setup
Jun 4 10:12:59 puppet pulpcore-api: self._wrapped = Settings(settings_module)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 157, in init
Jun 4 10:12:59 puppet pulpcore-api: mod = importlib.import_module(self.SETTINGS_MODULE)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
Jun 4 10:12:59 puppet pulpcore-api: return _bootstrap._gcd_import(name[level:], package, level)
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 994, in _gcd_import
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 971, in _find_and_load
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 955, in _find_and_load_unlocked
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 665, in _load_unlocked
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 678, in exec_module
Jun 4 10:12:59 puppet pulpcore-api: File “”, line 219, in _call_with_frames_removed
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/pulpcore/app/settings.py”, line 81, in
Jun 4 10:12:59 puppet pulpcore-api: plugin_app_config = entry_point.load()
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/pkg_resources/init.py”, line 2317, in load
Jun 4 10:12:59 puppet pulpcore-api: self.require(*args, **kwargs)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/pkg_resources/init.py”, line 2340, in require
Jun 4 10:12:59 puppet pulpcore-api: items = working_set.resolve(reqs, env, installer, extras=self.extras)
Jun 4 10:12:59 puppet pulpcore-api: File “/usr/lib/python3.6/site-packages/pkg_resources/init.py”, line 779, in resolve
Jun 4 10:12:59 puppet pulpcore-api: raise VersionConflict(dist, req).with_context(dependent_req)
Jun 4 10:12:59 puppet pulpcore-api: pkg_resources.ContextualVersionConflict: (PyYAML 5.4.1 (/usr/local/lib64/python3.6/site-packages), Requirement.parse(‘PyYAML<5.4.0,>=5.1.1’), {‘pulpcore’})
Jun 4 10:12:59 puppet pulpcore-api: [2021-06-04 10:12:59 -0400] [210369] [INFO] Worker exiting (pid: 210369)
Jun 4 10:12:59 puppet pulpcore-api: [2021-06-04 10:12:59 -0400] [210366] [INFO] Shutting down: Master
Jun 4 10:12:59 puppet pulpcore-api: [2021-06-04 10:12:59 -0400] [210366] [INFO] Reason: Worker failed to boot.
Jun 4 10:12:59 puppet systemd: pulpcore-worker@6.service holdoff time over, scheduling restart.
Jun 4 10:12:59 puppet systemd: pulpcore-api.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 4 10:12:59 puppet systemd: Unit pulpcore-api.service entered failed state.
Jun 4 10:12:59 puppet systemd: pulpcore-api.service failed.
Jun 4 10:12:59 puppet systemd: Stopped Pulp RQ Worker.
Jun 4 10:12:59 puppet systemd: Started Pulp RQ Worker.

8

In addition to the previous traceback from /var/log/messages I am seeing this when I attempt to use the pulpcore-manager command…

[root@puppet system]# pulpcore-manager --help
Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/pulpcore/app/settings.py”, line 269, in
CONTENT_ORIGIN
NameError: name ‘CONTENT_ORIGIN’ is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/bin/pulpcore-manager”, line 11, in
load_entry_point(‘pulpcore==3.9.1’, ‘console_scripts’, ‘pulpcore-manager’)()
File “/usr/lib/python3.6/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line
utility.execute()
File “/usr/lib/python3.6/site-packages/django/core/management/init.py”, line 357, in execute
django.setup()
File “/usr/lib/python3.6/site-packages/django/init.py”, line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File “/usr/lib/python3.6/site-packages/django/apps/registry.py”, line 122, in populate
app_config.ready()
File “/usr/lib/python3.6/site-packages/pulpcore/app/apps.py”, line 94, in ready
self.import_viewsets()
File “/usr/lib/python3.6/site-packages/pulpcore/app/apps.py”, line 127, in import_viewsets
from pulpcore.app.viewsets import NamedModelViewSet
File “/usr/lib/python3.6/site-packages/pulpcore/app/viewsets/init.py”, line 1, in
from .base import ( # noqa
File “/usr/lib/python3.6/site-packages/pulpcore/app/viewsets/base.py”, line 14, in
from pulpcore.openapi import PulpAutoSchema
File “/usr/lib/python3.6/site-packages/pulpcore/openapi/init.py”, line 25, in
from pulpcore.app.settings import INSTALLED_PULP_PLUGINS
File “/usr/lib/python3.6/site-packages/pulpcore/app/settings.py”, line 273, in
"CONTENT_ORIGIN is a required setting but it was not configured. This may be caused "
django.core.exceptions.ImproperlyConfigured: CONTENT_ORIGIN is a required setting but it was not configured. This may be caused by invalid read permissions of the settings file. Note that CONTENT_ORIGIN is set by the installer automatically.

9

NOTE: The permissions for settings.py are…

[root@puppet system]# ls -l “/usr/lib/python3.6/site-packages/pulpcore/app/settings.py”
-rw-r–r--. 1 root root 11320 Jan 21 14:30 /usr/lib/python3.6/site-packages/pulpcore/app/settings.py

10

Please use code blocs. They make it readable.

As for what’s wrong:

You appear to have packages installed locally that conflict. This is not supported. Note the /usr/local part.

11

Thanks for the response, I have uninstalled the python3 app ‘pulp-cli’ which I had installed to try to trouble-shoot this issue based on a suggestion I found while researching this problem.

12

I don’t know how to resolve the issue I am seeing but I think I found the cause.

When I run the pulpcore-manager command (as root or a non-root user) I get the traceback output I previously added to this issue.

If I run pulpcore-manager with the below variable set in my active environment I get the expected output.

I tried adding the variable to the .bashrc for the pulp user but it does not seem to get sourced when the pulp services are restarted so it adding it does not resolve the issue. I will try adding it to the service manifest file if I can figure out which service needs it.

PULP_SETTINGS=/etc/pulp/settings.py

13

Looks like I hit a dead-end…the environment variable is already being set in the service…so…I will shift to a new direction with this quest.

14

To clarify…the services that are failing to start are:

pulpcore-api.service
&
pulpcore-content.service

Both just stay activating; pulpcore-api.service shows this …

ExecStart=/usr/libexec/pulpcore/gunicorn pulpcore.app.wsgi:application -w 1 --access-logfile - (code=exited, status=3)

15

I believe the issue I am seeing is related to this
[Tasks stuck with "waiting for Pulp to start the task" - foreman-2.4 / Katello 4.0.0 / Pulp3]
with a slight variation.

I see the repo sync jobs waiting for days and never finishing but on my system doing curl https://puppet.ensco.win/pulp/api/v3/status never returns a response.

16

Thank you for pointing out the pulpcore conflict error. It appears that may have been the issue. I uninstalled the pulp-cli & PyYAML packages that I had installed to troubleshoot a sync issue. Now the pulpcore services will start. I believe this issue is resolved…problem was self-inflicted.

1 Like