Problem:
I realize this morning that pulpcore services are not running anymore on my Katello server
(demo effect while presenting composite content views to coworker… )
Foreman and Proxy versions:
Foreman 3.3 & Katello 4.5
Foreman and Proxy plugin versions:
foreman-tasks 6.0.2
foreman_puppet 4.0.1
foreman_remote_execution 7.1.1
katello 4.5.0 Distribution and version:
CentOS Linux release 7.9.2009 Other relevant data:
After a fews investigations I found that:
Aug 02 10:42:51 Updated: rh-postgresql12-postgresql-libs.x86_64 12.11-1.el7
Aug 02 10:42:51 Updated: rh-postgresql12-postgresql.x86_64 12.11-1.el7
Aug 02 10:42:57 Updated: rh-postgresql12-postgresql-server.x86_64 12.11-1.el7
Aug 02 10:42:57 Updated: rh-postgresql12-postgresql-server-syspaths.x86_64 12.11-1.el7
Aug 02 10:42:57 Updated: rh-postgresql12-postgresql-syspaths.x86_64 12.11-1.el7
Aug 03 12:13:25 Updated: puppet-agent.x86_64 6.28.0-1.el7
Aug 03 12:13:29 Updated: puppetserver.noarch 6.20.0-1.el7
Aug 03 12:13:29 Updated: tfm-rubygem-smart_proxy_remote_execution_ssh.noarch 0.7.1-1.fm3_3.el7
Aug 05 09:26:01 Updated: tfm-pulpcore-python3-jsonschema.noarch 4.6.0-1.el7
Aug 05 09:26:01 Updated: tfm-pulpcore-python3-django-readonly-field.noarch 1.1.1-1.el7
Aug 05 09:26:02 Updated: tfm-pulpcore-python3-pulp-rpm.noarch 3.17.7-1.el7
Aug 05 09:26:02 Updated: tfm-rubygem-foreman_remote_execution.noarch 7.1.1-1.fm3_3.el7
Could this updates have broke my pulpcore services ?
I just downgrade today’s updates and my services are back!
Hammer ping who wasn’t responding tells me everything looks fine, foreman-maintain health check too.
I just tried it, I’m getting errors during configuration steps (tried both with services running or not)
2022-08-05 15:15:23 [NOTICE] [configure] Starting system configuration.
2022-08-05 15:15:32 [NOTICE] [configure] 250 configuration steps out of 1927 steps complete.
2022-08-05 15:15:40 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2022-08-05 15:15:40 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2022-08-05 15:15:41 [NOTICE] [configure] 750 configuration steps out of 1937 steps complete.
2022-08-05 15:15:41 [NOTICE] [configure] 1000 configuration steps out of 1942 steps complete.
2022-08-05 15:15:44 [NOTICE] [configure] 1250 configuration steps out of 1943 steps complete.
2022-08-05 15:16:25 [NOTICE] [configure] 1500 configuration steps out of 1943 steps complete.
2022-08-05 15:16:39 [NOTICE] [configure] 1750 configuration steps out of 1943 steps complete.
2022-08-05 15:18:06 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:07 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Failed to call refresh: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:07 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-05 15:18:10 [NOTICE] [configure] System configuration has finished.
I downgrade again and rerun foreman-installer with no troubles and getting pulpcore services running back
I uncheck the solution on the second post because it’s more a workaround than a solution…
Founded this: [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: pkg_resources.DistributionNotFound: The 'importlib_resources>=1.4.0' distribution was not found and is required by jsonschema
And:
[DEBUG ] [configure] /Service[pulpcore-content.socket]: Skipping because of failed dependencies
[DEBUG ] [configure] /Stage[main]/Pulpcore::Apache/Pulpcore::Apache::Fragment[pulpcore]/Apache::Vhost::Fragment[pulpcore-http-pulpcore]/Concat::Fragment[foreman-pulpcore-http-pulpcore]/Concat_fragment[foreman-pulpcore-http-pulpcore]: Skipping because of failed dependencies
[DEBUG ] [configure] /Package[tfm-rubygem-smart_proxy_pulp]: Skipping because of failed dependencies
[DEBUG ] [configure] /Service[foreman-proxy]: Skipping because of failed dependencies
So I try python3 -m pip install importlib-resources
But I m still getting issues. I think there is missing dependencies that should have been embedded in the .rpm…
Im off of office for 2 weeks so I can’t keep working on it for now…
You have to check for those ERROR messages. You have posted DEBUG messages. The relevant errors should be next to those ERRORs.
Never, ever try to fulfill dependencies outside of the official repositories. Only evil can come from that… There is importlib-resources as rpm in the pulpcore repository. For 4.5 there should be version 5.4.0 installed.
I was experiencing this exact problem when I patched earlier yesterday. However, a timely updated package was released last night that has resolved the issue. The new package:
)
