Pulpcore services failed

GC29 · August 5, 2022, 12:31pm

Problem:
I realize this morning that pulpcore services are not running anymore on my Katello server
(demo effect while presenting composite content views to coworker… )

pulpcore-content.service failed
pulpcore-worker@1.service failed
pulpcore-worker@2.service failed

Expected outcome:
All services up and running

Foreman and Proxy versions:
Foreman 3.3 & Katello 4.5

Foreman and Proxy plugin versions:
foreman-tasks 6.0.2
foreman_puppet 4.0.1
foreman_remote_execution 7.1.1
katello 4.5.0
Distribution and version:
CentOS Linux release 7.9.2009
Other relevant data:

After a fews investigations I found that:

Aug 02 10:42:51 Updated: rh-postgresql12-postgresql-libs.x86_64 12.11-1.el7
Aug 02 10:42:51 Updated: rh-postgresql12-postgresql.x86_64 12.11-1.el7
Aug 02 10:42:57 Updated: rh-postgresql12-postgresql-server.x86_64 12.11-1.el7
Aug 02 10:42:57 Updated: rh-postgresql12-postgresql-server-syspaths.x86_64 12.11-1.el7
Aug 02 10:42:57 Updated: rh-postgresql12-postgresql-syspaths.x86_64 12.11-1.el7
Aug 03 12:13:25 Updated: puppet-agent.x86_64 6.28.0-1.el7
Aug 03 12:13:29 Updated: puppetserver.noarch 6.20.0-1.el7
Aug 03 12:13:29 Updated: tfm-rubygem-smart_proxy_remote_execution_ssh.noarch 0.7.1-1.fm3_3.el7
Aug 05 09:26:01 Updated: tfm-pulpcore-python3-jsonschema.noarch 4.6.0-1.el7
Aug 05 09:26:01 Updated: tfm-pulpcore-python3-django-readonly-field.noarch 1.1.1-1.el7
Aug 05 09:26:02 Updated: tfm-pulpcore-python3-pulp-rpm.noarch 3.17.7-1.el7
Aug 05 09:26:02 Updated: tfm-rubygem-foreman_remote_execution.noarch 7.1.1-1.fm3_3.el7

Could this updates have broke my pulpcore services ?

GC29 · August 5, 2022, 12:48pm

I just downgrade today’s updates and my services are back!
Hammer ping who wasn’t responding tells me everything looks fine, foreman-maintain health check too.

gvde · August 5, 2022, 1:00pm

Did you run foreman-installer after the update?

GC29 · August 5, 2022, 1:01pm

No

GC29 · August 5, 2022, 1:23pm

I just tried it, I’m getting errors during configuration steps (tried both with services running or not)

2022-08-05 15:15:23 [NOTICE] [configure] Starting system configuration.
2022-08-05 15:15:32 [NOTICE] [configure] 250 configuration steps out of 1927 steps complete.
2022-08-05 15:15:40 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2022-08-05 15:15:40 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2022-08-05 15:15:41 [NOTICE] [configure] 750 configuration steps out of 1937 steps complete.
2022-08-05 15:15:41 [NOTICE] [configure] 1000 configuration steps out of 1942 steps complete.
2022-08-05 15:15:44 [NOTICE] [configure] 1250 configuration steps out of 1943 steps complete.
2022-08-05 15:16:25 [NOTICE] [configure] 1500 configuration steps out of 1943 steps complete.
2022-08-05 15:16:39 [NOTICE] [configure] 1750 configuration steps out of 1943 steps complete.
2022-08-05 15:18:06 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:07 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Failed to call refresh: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:07 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:10 [NOTICE] [configure] System configuration has finished.

GC29 · August 5, 2022, 1:32pm

I downgrade again and rerun foreman-installer with no troubles and getting pulpcore services running back
I uncheck the solution on the second post because it’s more a workaround than a solution…

gvde · August 5, 2022, 1:45pm

Check the installer log for details on the error.

GC29 · August 5, 2022, 2:23pm

Founded this:
[DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: pkg_resources.DistributionNotFound: The 'importlib_resources>=1.4.0' distribution was not found and is required by jsonschema

And:

[DEBUG ] [configure] /Service[pulpcore-content.socket]: Skipping because of failed dependencies
[DEBUG ] [configure] /Stage[main]/Pulpcore::Apache/Pulpcore::Apache::Fragment[pulpcore]/Apache::Vhost::Fragment[pulpcore-http-pulpcore]/Concat::Fragment[foreman-pulpcore-http-pulpcore]/Concat_fragment[foreman-pulpcore-http-pulpcore]: Skipping because of failed dependencies
[DEBUG ] [configure] /Package[tfm-rubygem-smart_proxy_pulp]: Skipping because of failed dependencies
[DEBUG ] [configure] /Service[foreman-proxy]: Skipping because of failed dependencies

So I try
python3 -m pip install importlib-resources
But I m still getting issues. I think there is missing dependencies that should have been embedded in the .rpm…

Im off of office for 2 weeks so I can’t keep working on it for now…

gvde · August 5, 2022, 2:33pm

You have to check for those ERROR messages. You have posted DEBUG messages. The relevant errors should be next to those ERRORs.

Never, ever try to fulfill dependencies outside of the official repositories. Only evil can come from that… There is importlib-resources as rpm in the pulpcore repository. For 4.5 there should be version 5.4.0 installed.

Flyhalf205 · August 5, 2022, 6:08pm

Getting pretty much same thing looks like the OP is getting. I just attempted upgrading from 3.2 to 3.3.

2022-08-05 12:45:10 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput'
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Traceback (most recent call last):
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     ws.require(__requires__)
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     needed = self.resolve(parse_requirements(requirements))
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     raise VersionConflict(dist, req).with_context(dependent_req)
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: pkg_resources.ContextualVersionConflict: (importlib-resources 0.0.0 (/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages), Requirement.parse('importlib_resources>=1.4.0'), {'jsonschema'})
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: During handling of the above exception, another exception occurred:
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Traceback (most recent call last):
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/theforeman/tfm-pulpcore/root/usr/bin/pulpcore-manager", line 6, in <module>
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     from pkg_resources import load_entry_point
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3252, in <module>
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     def _initialize_master_working_set():
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     f(*args, **kwargs)
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     working_set = WorkingSet._build_master()
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 585, in _build_master
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return cls._build_from_requirements(__requires__)
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     dists = ws.resolve(reqs, Environment())
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     raise DistributionNotFound(req, requirers)
2022-08-05 12:45:10 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: pkg_resources.DistributionNotFound: The 'importlib_resources>=1.4.0' distribution was not found and is required by jsonschema
2022-08-05 12:45:10 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 12:45:10 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

gvde · August 5, 2022, 7:39pm

What version of the importlib-resources rpm do you have installed? I don’t know the full name right now. It contains importlib in the name…

Flyhalf205 · August 5, 2022, 7:41pm

rpm -qa | grep importlib

python3-importlib-metadata-1.7.0-1.el7.noarch
tfm-pulpcore-python3-importlib-resources-5.4.0-2.el7.noarch
tfm-pulpcore-python3-importlib-metadata-4.10.1-2.el7.noarch