Salt Proxy State Import Problem

joni1993 · September 3, 2019, 3:02pm

Problem:
I’ve setup salt+salt-proxy and everything works fine (highstate reports, grains colletion) but i cant import the state from the salt-master because of

ERF12-4701 [ProxyAPI::ProxyException]: Unable to fetch Salt states list ([RestClient::NotFound]: 404 Not Found) for proxy https://foremanproxyaddr:8443/salt/

If i hit the import button i can see some successfull transfer in /var/log/foreman-proxy/proxy.log:

2019-09-03T16:56:06  [D] accept: 2001:470:7751::12:54954
2019-09-03T16:56:06  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:06 ef441d66 [I] Started GET /environments 
2019-09-03T16:56:08 ef441d66 [I] Finished GET /environments with 200 (1877.44 ms)
2019-09-03T16:56:08  [D] close: 2001:470:7751::12:54954
2019-09-03T16:56:08  [D] accept: 2001:470:7751::12:54956
2019-09-03T16:56:08  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:08 ef441d66 [I] Started GET /environments/2016.12 
2019-09-03T16:56:09 ef441d66 [I] Finished GET /environments/2016.12 with 200 (1112.93 ms)
2019-09-03T16:56:09  [D] close: 2001:470:7751::12:54956
2019-09-03T16:56:09  [D] accept: 2001:470:7751::12:54958
2019-09-03T16:56:09  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:09 ef441d66 [I] Started GET /environments/2016.12.1 
2019-09-03T16:56:10 ef441d66 [I] Finished GET /environments/2016.12.1 with 200 (769.19 ms)
2019-09-03T16:56:10  [D] close: 2001:470:7751::12:54958
2019-09-03T16:56:10  [D] accept: 2001:470:7751::12:54960
2019-09-03T16:56:10  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:10 ef441d66 [I] Started GET /environments/2017.2 
2019-09-03T16:56:11 ef441d66 [I] Finished GET /environments/2017.2 with 200 (775.43 ms)
2019-09-03T16:56:11  [D] close: 2001:470:7751::12:54960
2019-09-03T16:56:11  [D] accept: 2001:470:7751::12:54962
2019-09-03T16:56:11  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:11 ef441d66 [I] Started GET /environments/2017.2.1 
2019-09-03T16:56:12 ef441d66 [I] Finished GET /environments/2017.2.1 with 200 (1164.63 ms)
2019-09-03T16:56:12  [D] close: 2001:470:7751::12:54962
2019-09-03T16:56:12  [D] accept: 2001:470:7751::12:54964
2019-09-03T16:56:12  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:12 ef441d66 [I] Started GET /environments/2017.2.2 
2019-09-03T16:56:13 ef441d66 [I] Finished GET /environments/2017.2.2 with 200 (822.19 ms)
2019-09-03T16:56:13  [D] close: 2001:470:7751::12:54964
2019-09-03T16:56:13  [D] accept: 2001:470:7751::12:54966
2019-09-03T16:56:13  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:13 ef441d66 [I] Started GET /environments/base 
2019-09-03T16:56:14 ef441d66 [I] Finished GET /environments/base with 200 (791.02 ms)
2019-09-03T16:56:14  [D] close: 2001:470:7751::12:54966
2019-09-03T16:56:14  [D] accept: 2001:470:7751::12:54968
2019-09-03T16:56:14  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:14 ef441d66 [I] Started GET /environments/certbot 
2019-09-03T16:56:15 ef441d66 [I] Finished GET /environments/certbot with 200 (808.8 ms)
2019-09-03T16:56:15  [D] close: 2001:470:7751::12:54968
2019-09-03T16:56:15  [D] accept: 2001:470:7751::12:54970
2019-09-03T16:56:15  [D] Rack::Handler::WEBrick is invoked.
2019-09-03T16:56:15 ef441d66 [I] Started GET /environments/feature/spm-packaging-meta 
2019-09-03T16:56:15 ef441d66 [I] Finished GET /environments/feature/spm-packaging-meta with 404 (1.64 ms)
2019-09-03T16:56:15  [D] close: 2001:470:7751::12:54970

but on the foreman host it throws an error:

ProxyAPI::ProxyException: ERF12-4701 [ProxyAPI::ProxyException]: Unable to fetch Salt states list ([RestClient::NotFound]: 404 Not Found) for proxy https://foremanproxyaddr:8443/salt/
/usr/share/foreman/vendor/ruby/2.5.0/gems/foreman_salt-11.0.0/app/lib/proxy_api/salt.rb:43:in `rescue in states_list'
/usr/share/foreman/vendor/ruby/2.5.0/gems/foreman_salt-11.0.0/app/lib/proxy_api/salt.rb:34:in `states_list'
/usr/share/foreman/vendor/ruby/2.5.0/gems/foreman_salt-11.0.0/app/controllers/foreman_salt/state_importer.rb:16:in `fetch_states_from_proxy'
/usr/share/foreman/vendor/ruby/2.5.0/gems/foreman_salt-11.0.0/app/controllers/foreman_salt/salt_modules_controller.rb:61:in `import'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/apipie-rails-0.5.16/lib/apipie/static_dispatcher.rb:66:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/apipie-rails-0.5.16/lib/apipie/extractor/recorder.rb:137:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/apipie-rails-0.5.16/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:25:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/secure_headers-6.1.0/lib/secure_headers/middleware.rb:11:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
/usr/share/foreman/vendor/ruby/2.5.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Foreman and Proxy versions:
Foreman 1.22
Foreman Proxy 1.22.1
Salt Version 2.1.9

Salt Version: 2019.2.0 (Fluorine)

I set all the certs and options according to the guide on Foreman :: Plugin Manuals

Bernhard_Suttner · October 22, 2019, 1:39pm

foreman_salt tries to fetch the states of each salt environment.

The smart proxy salt will use the Rest API of salt to receive the environment and the salt states. You can run the following commands on the salt master to see whats actually going on:

#returns the list of environments
salt-run fileserver.envs 

#returns all available salt states
salt-run fileserver.file_list

# returns the salt state of the env "base"
salt-run fileserver.file_list saltenv=base

joni1993 · October 22, 2019, 2:31pm

Thank you for your answer, I already figured out the problem and I think this is a bug in Salt itself (either salt master or salt-api)
I used several gitfs formulas which by default use git branches for environments. This seems to work most of the time but it fails for the branch feature/spm-packaging-meta from the letsencrypt formula https://github.com/salt-formulas/salt-formula-letsencrypt/branches.
Probably salt-api fails to translate the slash in the branch name when called for listing of the environment.

Also thank you for your Froscon talk! Is there any schedule when the new push based system is ready for release?

Bernhard_Suttner · October 23, 2019, 7:22am

Great to hear that its not a foreman_salt issue and thank you very much for the explanation.

Regading the “push based” system, you mean:
https://github.com/theforeman/smart_proxy_salt/pull/39
Its ready, when I push the “merge” button But it would be great to have feedback from users first. Therefore, it would be great if you could test it. If you need help, just let me know.

joni1993 · October 23, 2019, 8:12am

I will try to test the PR and get back to you. It may take some time because i already fail at getting remote execution to work (manual commands, “run salt” works strangely).
This PR does not account for the “instant feedback in foreman when running highstate” if i see this correctly, as you mentioned it in the talk.