Installed a new smart proxy using Foreman 2.5.2 and Katello 4.1.2
Smart proxy is registered. Comm status is green. 14 active features.
However in overview on the right for Pulp Storage I get:
Oops, we're sorry but something went wrong Not all necessary pulp workers running at https://foremanproxy02.fishy.com/pulp/api/v3.
Main Foreman/Katello server (still on 2.5.1 / 4.1.0)
# foreman-rake errors:fetch_log request_id=2279112c
`/usr/share/foreman` is not writable.
Bundler will use `/tmp/bundler20210802-9205-1l3gg859205' as your home directory temporarily.
2021-08-02T10:07:06 [I|app|2279112c] Started GET "/smart_proxies/3-foremanproxy02-fishy-com/pulp_storage" for 10.50.147.253 at 2021-08-02 10:07:06 +0000
2021-08-02T10:07:06 [I|app|2279112c] Processing by SmartProxiesController#pulp_storage as HTML
2021-08-02T10:07:06 [I|app|2279112c] Parameters: {"id"=>"3-foremanproxy02-fishy-com"}
2021-08-02T10:07:07 [W|app|2279112c] Not all necessary pulp workers running at https://foremanproxy02.fishy.com/pulp/api/v3.
2021-08-02T10:07:07 [I|app|2279112c] Backtrace for 'Not all necessary pulp workers running at https://foremanproxy02.fishy.com/pulp/api/v3.' error (RuntimeError): Not all necessary pulp workers running at https://foremanproxy02.fishy.com/pulp/api/v3.
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/models/katello/ping.rb:201:in `pulp3_without_auth'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/models/katello/concerns/smart_proxy_extensions.rb:370:in `ping_pulp3'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/models/katello/concerns/smart_proxy_extensions.rb:169:in `pulp_disk_usage'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/controllers/katello/concerns/smart_proxies_controller_extensions.rb:29:in `pulp_storage'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in `process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
2279112c | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
2279112c | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
2279112c | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in `process'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in `process'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in `dispatch'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in `dispatch'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `each'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `serve'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/lib/katello/middleware/event_daemon.rb:10:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
2279112c | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
2279112c | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in `run_callbacks'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
2279112c | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/lib/katello/prevent_json_parsing.rb:12:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
2279112c | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-08-02T10:07:07 [I|app|2279112c] Rendering common/500.html.erb
2021-08-02T10:07:07 [I|app|2279112c] Rendered common/500.html.erb (Duration: 6.9ms | Allocations: 6901)
2021-08-02T10:07:07 [I|app|2279112c] Completed 500 Internal Server Error in 978ms (Views: 7.6ms | ActiveRecord: 4.7ms | Allocations: 28344)
Foreman version: 2.5.1
Plugins:
- foreman-tasks 4.1.2
- foreman_ansible 6.3.0
- foreman_openscap 4.3.2
- foreman_remote_execution 4.5.0
- katello 4.1.0
And to upgrade my main server I’m hitting this dependency issue
I have something similar when installing Foreman 2.5.2 and Katello 4.1.2 on CentOS 8 test server. Not sure what the exact error message was, but it happened when creating first repository.
I was required to manually start remaining workers and after that all went flawlessly.
systemctl start pulpcore-worker@2
On CentOS 7.9
2 pulpcore workers are running but the pulp storage in overview still shows the error…
18972 ? Ss 0:04 /usr/bin/python3 /usr/bin/pulpcore-worker
19038 ? Ss 0:04 /usr/bin/python3 /usr/bin/pulpcore-worker
# systemctl status pulpcore-worker@1
● pulpcore-worker@1.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-08-02 09:38:46 UTC; 3h 19min ago
Main PID: 18972 (pulpcore-worker)
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
└─18972 /usr/bin/python3 /usr/bin/pulpcore-worker
Aug 02 09:38:46 foremanproxy02.fishy.com systemd[1]: Started Pulp Worker.
Aug 02 09:38:50 foremanproxy02.fishy.com pulpcore-worker-1[18972]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 02 09:38:50 foremanproxy02.fishy.com pulpcore-worker-1[18972]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '18972@foremanproxy02.fishy.com' discovered
Hint: Some lines were ellipsized, use -l to show in full.
# systemctl status pulpcore-worker@2
● pulpcore-worker@2.service - Pulp Worker
Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-08-02 09:38:47 UTC; 3h 19min ago
Main PID: 19038 (pulpcore-worker)
CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
└─19038 /usr/bin/python3 /usr/bin/pulpcore-worker
Aug 02 09:38:47 foremanproxy02.fishy.com systemd[1]: Started Pulp Worker.
Aug 02 09:38:51 foremanproxy02.fishy.com pulpcore-worker-2[19038]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Aug 02 09:38:51 foremanproxy02.fishy.com pulpcore-worker-2[19038]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '19038@foremanproxy02.fishy.com' discovered
Hint: Some lines were ellipsized, use -l to show in full.
And I have upgraded a working smart proxy and now shows the same pulp storage issue… Blimey 
Could that be a side effect of having Main Foreman/Katello server on 2.5.1/4.1.0 and the 2x external smart proxies on 2.5.2/4.1.2 ?
Is there something I can check to give more info?
Hey Fred,
I’m curious what exactly is the status that’s getting reported back from the proxy to the Katello server. You can find that by doing this from foreman-rake console on the Katello server:
SmartProxy.all
From the list of proxies, note the ‘id’ field of the proxy that’s having the problem
then do:
Katello::Ping.pulp3_without_auth(SmartProxy.find($ID_YOU_FOUND).pulp3_url)
Obviously you can omit any hostnames you with to remain private from the data you share.
Hi Jonathon
For my newest external foreman proxy
# foreman-rake console
`/usr/share/foreman` is not writable.
Bundler will use `/tmp/bundler20210803-4783-eoj6t4783' as your home directory temporarily.
Loading production environment (Rails 6.0.3.7)
irb(main):001:0> Katello::Ping.pulp3_without_auth(SmartProxy.find(3).pulp3_url)
Traceback (most recent call last):
3: from lib/tasks/console.rake:5:in `block in <top (required)>'
2: from (irb):1
1: from katello (4.1.0) app/models/katello/ping.rb:201:in `pulp3_without_auth'
RuntimeError (Not all necessary pulp workers running at https://foremanproxy02.fishy.com/pulp/api/v3.)
For my external foreman proxy that was just upgraded
irb(main):002:0> Katello::Ping.pulp3_without_auth(SmartProxy.find(2).pulp3_url)
Traceback (most recent call last):
4: from lib/tasks/console.rake:5:in `block in <top (required)>'
3: from (irb):1
2: from (irb):2:in `rescue in irb_binding'
1: from katello (4.1.0) app/models/katello/ping.rb:201:in `pulp3_without_auth'
RuntimeError (Not all necessary pulp workers running at https://foremanproxy01.fishy.com/pulp/api/v3.)
For the main Foreman/Katello proxy
irb(main):003:0> Katello::Ping.pulp3_without_auth(SmartProxy.find(1).pulp3_url)
=> {"versions"=>[{"component"=>"core", "version"=>"3.11.2"}, {"component"=>"rpm", "version"=>"3.11.0"}, {"component"=>"file", "version"=>"1.6.0"}, {"component"=>"deb", "version"=>"2.11.2"}, {"component"=>"container", "version"=>"2.5.3"}, {"component"=>"certguard", "version"=>"1.2.0"}, {"component"=>"ansible", "version"=>"0.7.3"}], "onli
... and lots more
All fixed after upgrading main Foreman/Katello server to 2.5.2/4.1.2 (same as my 2x external smart proxies). Tfm-rubygem-katello-4.1.2-1.el7.noarch dependency - #8 by fred_demarcy
I missed the critical part of your earlier message - that the Katello server was on a version behind the Smart Proxy. That absolutely explains the problem, and I’m glad to hear upgrading fixed the problem for you. We support running the smart proxy at $VERSION-1 compared to the Katello server, but not the other way around.
No worries. I was suspecting this was the case. Was just waiting for the repo to be fixed to verify that was the case.