vCenter Storage error 500 when creating new hosts

Problem:

When trying to create a host to be deployed on vCenter, the Storage section of the Virtual Machine tab shows error 500

Expected outcome:

It should present Data Store and Disk Mode dropdown menu to choose from the options available in the vCenter storage.

Foreman and Proxy versions:

1.22.0

Foreman and Proxy plugin versions:

1.22.0

Distribution and version:

Ubuntu 16.04, vCenter Server 6.7

Other relevant data:

This happens sometimes when creating a host to be deployed on vCenter. It’s not always reproducible; when it happens, a few refreshes on the web UI and it works again. Here’s a partial log from Foreman:

2019-11-04T18:30:10 [I|app|d144ddb3] Started GET "/api/v2/compute_resources/1-vCenter/available_storage_domains?cluster_id=XXXX" for XXXXXXXXXX at 2019-11-04 18:30:10 -0300
2019-11-04T18:30:10 [I|app|d144ddb3] Processing by Api::V2::ComputeResourcesController#available_storage_domains as JSON
2019-11-04T18:30:10 [I|app|d144ddb3]   Parameters: {"cluster_id"=>"XXXX", "apiv"=>"v2", "id"=>"1-vCenter"}
2019-11-04T18:30:10 [I|app|d144ddb3] Current user set to XXXXXXXXXXXXX
2019-11-04T18:30:10 [W|app|d144ddb3] Action failed
 | NameError: uninitialized constant Fog::Vsphere::Compute::Datastore
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:6:in `load'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:114:in `datastores'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:188:in `available_storage_domains'
 | /usr/share/foreman/app/controllers/api/v2/compute_resources_controller.rb:169:in `available_storage_domains'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.16/lib/apipie/static_dispatcher.rb:66:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.16/lib/apipie/extractor/recorder.rb:137:in `call'
 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.16/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:25:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/secure_headers-6.1.1/lib/secure_headers/middleware.rb:11:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
 | /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
 | /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
 | /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
 | /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
 | /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
 | /usr/share/foreman/vendor/ruby/2.3.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-11-04T18:30:10 [I|app|d144ddb3]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-11-04T18:30:10 [I|app|d144ddb3]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (1.2ms)
2019-11-04T18:30:10 [I|app|d144ddb3] Completed 500 Internal Server Error in 62ms (Views: 12.3ms | ActiveRecord: 6.8ms)

Exact same problem here

The workaround, suggested by one of Foreman devs, is to disable caching on vCenter compute resource (Infrastructure>Compute Resources>edit your vCenter instance>uncheck Enable Caching).

After that, this issue goes away. I think it has something to do with caching on vCenter still being an unstable feature by that time. Maybe upgrading to a more recent Foreman version will solve the issue for good.

Hope it helps!

Thnak you it obviously resolve my issue even though I have the latest version of Foreman 1.24.2 maybe I can test the newest version 2.0

This occurs in 2.0 as well. Even though I only see the error in Webinterface when creating a new host. Creation still works though.