Issue after yum update -> Foreman 2.5.2 -pulpcore_workers failing

Support
1

I was running Foreman 2.5.1 - Katello 4.1, the last update run changed that to Foreman 2.5.2. foreman-installer ran without problems but now not all Foreman services are started.
foreman-maintain health check reports:
Check whether all services are running using the ping call: [FAIL]
Couldn’t connect to the server: undefined method `to_sym’ for nil:NilClass
Seems to be related to the pulpcore_worker services, those are in a failed state and the GUI shows for the pulp storage an error message:
Not all necessary pulp workers running at https://<foreman_server>/pulp/api/v3.

This is a blocking issue - any help would be greatly appreciated

2

Can confirm the same error at my end. here is the log from my side:

# foreman-rake errors:fetch_log request_id=eb955f14
`/usr/share/foreman` is not writable.
Bundler will use `/tmp/bundler20210716-42083-14djhbr42083' as your home directory temporarily.
2021-07-16T14:19:49 [I|app|eb955f14] Started GET "/smart_proxies/5-p5-fr4-fmpxy-1-ibfr4eq-local/pulp_storage" for 10.116.36.148 at 2021-07-16 14:19:49 +0000
2021-07-16T14:19:49 [I|app|eb955f14] Processing by SmartProxiesController#pulp_storage as HTML
2021-07-16T14:19:49 [I|app|eb955f14]   Parameters: {"id"=>"5-p5-fr4-fmpxy-1-ibfr4eq-local"}
2021-07-16T14:19:49 [W|app|eb955f14] Not all necessary pulp workers running at https://p5-fr4-fmpxy-1.ibfr4eq.local/pulp/api/v3.
2021-07-16T14:19:49 [I|app|eb955f14] Backtrace for 'Not all necessary pulp workers running at https://p5-fr4-fmpxy-1.ibfr4eq.local/pulp/api/v3.' error (RuntimeError): Not all necessary pulp workers running at https://p5-fr4-fmpxy-1.ibfr4eq.local/pulp/api/v3.
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/models/katello/ping.rb:201:in `pulp3_without_auth'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/models/katello/concerns/smart_proxy_extensions.rb:370:in `ping_pulp3'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/models/katello/concerns/smart_proxy_extensions.rb:169:in `pulp_disk_usage'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/controllers/katello/concerns/smart_proxies_controller_extensions.rb:29:in `pulp_storage'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in `process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 eb955f14 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 eb955f14 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 eb955f14 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in `process'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in `process'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in `dispatch'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in `dispatch'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `each'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/lib/katello/middleware/event_daemon.rb:10:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 eb955f14 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 eb955f14 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 eb955f14 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in `run_callbacks'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 eb955f14 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/lib/katello/prevent_json_parsing.rb:12:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
 eb955f14 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-07-16T14:19:49 [I|app|eb955f14]   Rendering common/500.html.erb
2021-07-16T14:19:49 [I|app|eb955f14]   Rendered common/500.html.erb (Duration: 2.0ms | Allocations: 795)
2021-07-16T14:19:49 [I|app|eb955f14] Completed 500 Internal Server Error in 350ms (Views: 3.1ms | ActiveRecord: 89.0ms | Allocations: 141680)

Foreman version: 2.5.1
Plugins: 
 - foreman-tasks 4.1.2
 - foreman_chef 0.10.0
 - foreman_remote_execution 4.5.1
 - katello 4.1.0
3

Can you check the status of the pulpcore-worker services, e.g.:
systemctl status pulpcore-worker@2.service

I noticed some problem with that unit - just want to confirm first if your workers are in the failed state too

4

yes. it’s down

# systemctl status pulpcore-worker@2.service -l
● pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2021-07-16 14:34:46 UTC; 2s ago
  Process: 84411 ExecStart=/usr/libexec/pulpcore/pulpcore-worker (code=exited, status=203/EXEC)
 Main PID: 84411 (code=exited, status=203/EXEC)

Jul 16 14:34:46 p5-fr4-fmpxy-1 systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=203/EXEC
Jul 16 14:34:46 p5-fr4-fmpxy-1 systemd[1]: Unit pulpcore-worker@2.service entered failed state.
Jul 16 14:34:46 p5-fr4-fmpxy-1 systemd[1]: pulpcore-worker@2.service failed.
5

here is explanation from systemd

systemd[85247]: Failed at step EXEC spawning /usr/libexec/pulpcore/pulpcore-worker: No such file or directory
6

Right - so check the ExecStart in /etc/systemd/system/pulpcore-worker@.service
if it shows:
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
Change it to (after making a backup copy):
ExecStart=/usr/libexec/pulpcore/rq worker
-w pulpcore.tasking.worker.PulpWorker
-c ‘pulpcore.rqconfig’
–disable-job-desc-logging

There is no ‘/usr/libexec/pulpcore/pulpcore-worker’ - I got that ExecStart from a proxy that is not upgraded yet. After the change run systemctl daemon-reload

7

well, that makes the pulpcore-worker processes start again but I a service restart still results in:
…

  • All services displayed [FAIL]
    Some services are not running (pulpcore-worker@*)
8

yep. no luck. someone forgot to update systemd file in that commit probably.
I assume they should fix it over the weekend.

9

It looks to me as if 2.5.2 is supposed to work with pulp 3.14 which will come with katello 4.1.1 or 4.2 which isn’t available, yet…

Or it seems 4.1.1 has just arrived in the repository. Wasn’t there this morning…

10

Ah could be. There was a couple of versions ago some bug, also caused by foreman and katello not being synced in their releases.

11

OK, so we just have to wait until everything is in sync, repo-wise?

12

It already is - running the update as we speak (don’t forget to revert the changes I mentioned above aka restore the backup file)

13

Yes, you must upgrade Katello as well.

Originally we talked about adding an RPM conflicts in the installer but that was forgotten which is breaking installations.

14

Maybe its a race condition where this Katello package was supposed to be installed before Foreman, but after upgrade now to katello 4.1.1, no changes. pulpcore worker is still down for me with the same issue.

15

Hmm - that is odd - working for me now - you did revert the unit change and performed a daemon-reload?

16

yep. how does your systemd file look now?

17

The update did not touch it so I reverted manually

18

Do you use the original repo files or do you sync from your katello server? From Katello 4.1.0 to 4.1.1 the repository file now contains pulp 3.14 not 3.11 anymore…

And don’t forget to run foreman-installer after the update. That should create the correct pulp-worker systemd service.

19

My server syncs from the public internet so the original repo files and it did get pulp 3.14:

rpm -qa | grep pulp | grep 3.14

tfm-rubygem-pulpcore_client-3.14.1-1.el7.noarch
python3-pulpcore-3.14.1-1.el7.noarch

20

yep, I’m still missing that file /usr/libexec/pulpcore/pulpcore-worker
I ran yum update and foreman-installer afterwards, standard. But did not help.

Centos and epel repo are synced from internet daily and hosted inside network.
foreman/katello I pull directly from the internet, offical repos.

I’ll pull those, installed pulp version is still 3.11. Tnx for your help.