Foreman 1.20.2 / VMware - Sporadic HTTP 500 when querying /api/compute_resources API

Sorry, there was a mistake in my first post. The sporadic error output of the REST query is actually “uninitialized constant Fog::Compute::Vsphere::Cluster”:

$ curl --user *** -k -X GET -H "Content-Type:application/json" -H "Accept:application/json" https://foreman.***.de/api/compute_resources/4 | jq
Enter host password for user '***':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    83    0    83    0     0     26      0 --:--:--  0:00:03 --:--:--    26
{
  "error": {
    "message": "uninitialized constant Fog::Compute::Vsphere::Cluster"
  }
}

I hope this makes more sense now … The rest of my observations remain unchanged.

production.log:

2019-05-15T01:01:50 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 01:01:50 +0200
2019-05-15T01:01:50 [I|app|d8043] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T01:01:50 [I|app|d8043]   Parameters: {"apiv"=>"v2", "id"=>"4", "compute_resource"=>{}}
2019-05-15T01:01:51 [I|app|] Started GET "/notification_recipients" for *** at 2019-05-15 01:01:51 +0200
2019-05-15T01:01:51 [I|app|2dfd1] Processing by NotificationRecipientsController#index as JSON
2019-05-15T01:01:51 [I|app|2dfd1] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|2dfd1] Completed 200 OK in 6ms (Views: 0.2ms | ActiveRecord: 0.6ms)
2019-05-15T01:01:51 [I|app|d8043] Authorized user ***(***)
2019-05-15T01:01:51 [I|app|d8043] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|d8043]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T01:01:51 [I|app|d8043]   Rendered api/v2/compute_resources/show.json.rabl (22.6ms)
2019-05-15T01:01:51 [W|app|d8043] Action failed
| ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:158:in `available_clusters'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:627:in `normalize_vm_attrs'
| /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl__3693622661913919226_70129623652120'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
| /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
| /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
| /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
| /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T01:01:51 [I|app|d8043]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T01:01:51 [I|app|d8043]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.3ms)

It actually doesn’t :laughing: I gave myself decent time to think about it, and nothing came up.

I really have no idea what could be the cause here, so I can try to guide you through some debugging process, to try figure it out.

You could try to add these at the begging of the method, in which it all goes down.

# app/models/compute_resources

#....

def available_clusters
  require 'fog/vsphere/version'
  Rails.logger.info "current vsphere version is: #{Fog::Vsphere::VERSION}"
  Rails.logger.info "Fog::Compute is #{defined? Fog::Compute}"
  Rails.logger.info "Fog::Compute::Vsphere::Cluster is #{defined? Fog::Compute::Vsphere::Cluster}"
  #...
end

What should give us in the logs answers for:

  • Your code is not reloading fog-vsphere on the fly for another version
  • Fog is not unloaded in between the requests

If everything is all right, I guess we will have to examine the cache for it’s raw value.

Hi,

I’ve added your debug code to the available_clusters method (/usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb)

This is how a successful request looks like:

2019-05-15T18:11:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:11:36 +0200
2019-05-15T18:11:36 [I|app|ef15d] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:11:36 [I|app|ef15d]   Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:11:36 [I|app|ef15d] Current user set to *** (admin)
2019-05-15T18:11:36 [I|app|ef15d]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:11:36 [I|app|ef15d] current vsphere version is: 2.3.0
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute is constant
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute::Vsphere::Cluster is constant
2019-05-15T18:11:36 [I|app|ef15d]   Rendered api/v2/compute_resources/show.json.rabl (44.2ms)
2019-05-15T18:11:36 [I|app|ef15d] Completed 200 OK in 53ms (Views: 44.1ms | ActiveRecord: 2.4ms)

This was an unsuccessful query:

2019-05-15T18:06:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:06:36 +0200
2019-05-15T18:06:36 [I|app|819fb] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:06:36 [I|app|819fb]   Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:06:36 [I|app|819fb] Current user set to *** (admin)
2019-05-15T18:06:36 [I|app|819fb]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:06:36 [I|app|819fb] current vsphere version is: 2.3.0
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute is constant
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute::Vsphere::Cluster is 
2019-05-15T18:06:36 [I|app|819fb]   Rendered api/v2/compute_resources/show.json.rabl (24.2ms)
2019-05-15T18:06:36 [W|app|819fb] Action failed
 | ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:162:in `available_clusters'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:631:in `normalize_vm_attrs'
 | /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl___541234303595413005_70067119553160'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
 | /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
 | /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T18:06:36 [I|app|819fb]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T18:06:36 [I|app|819fb]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.4ms)
2019-05-15T18:06:36 [I|app|819fb] Completed 500 Internal Server Error in 35ms (Views: 1.1ms | ActiveRecord: 2.3ms)

Cheers,
Martin

Yesterday I’ve updated another Foreman server to 1.20.2 which is now showing the same issue (sporadic HTTP 500 / “uninitialized constant Fog::Compute::Vsphere::Cluster”. This Foreman Server uses the same VSphere Server to create hosts as the other one.

Can you please try to do a “Refresh Cache” on the “Infrastructure -> Compute Resources -> Your Vmware Provider” page?

Does it work afterwards? If this is the case, does it fail again?

In my experience it works after refreshing the cache, but it does start to fail again after a short while. The only workaround I have right now is to disable the compute resource cache completely, which results in very slow host edit pages.

I’m having similar problems on 1.21.3, but I believe it happened for me on 1.20 as well.

Typically it happens when trying to edit a host in the GUI. Now and then we get the error message:

Oops, we're sorry but something went wrong uninitialized constant Fog::Compute::Vsphere::Network Did you mean? Fog::Network

We then just try again and then it works at the second or third attempt. I just did a ‘refresh cache’ on my vmware compute resource, but that didn’t help.

We’re on RHEL7 with standard rpm installation, standalone installation (no HA, no LB)
tfm-rubygem-fog-vsphere-2.5.0-1.el7.noarch

I believe it is something about rails autoloading vs passanger and fog nasty namespaces. I wonder though how comes we are hitting those now out of the blue :thinking:
I had blew too much time on trying to fix that and there is recently introduced simple patch by, which should have fix that:

That is not rooting out the evil, but it is solving the issue, could you guys test it, if it works for you?

YES, that commit seems to have fixed the issue in my environment.
Good enough for me, thank you!

Hi,

I’m still getting sporadic HTTP 500 errors after applying the patch (same REST query, same error frequency). However the error message is now a different one:

ActionView::Template::Error: undefined method `resource_pools’ for nil:NilClass

production.log snippet:

production_log_snippet.log (20.5 KB)

Greetings,
Martin

I also get another, similar error messages in production.log, but not as frequent. I got these before the patch as well, but I don’t know exactly in what situation since it to my knowledge hasnt affected any users.

Error messages like:

2019-05-28T05:50:23 [W|app|b2706f9c] Action failed
 | NameError: uninitialized constant Fog::Compute::Vsphere::StoragePod
 | Did you mean?  Fog::Storage

Could you guys please try to add

# app/models/concerns/fog_extensions.rb:54
require 'fog/vsphere/compute'

and try again?

I already have that line on 1.21

the line is already included (1.20.2):

 54 if Foreman::Model::Vmware.available?
 55   require 'fog/vsphere'
 56   require 'fog/vsphere/models/compute/cluster'
 57   require 'fog/vsphere/models/compute/network'
 58   require 'fog/vsphere/compute'
 59   require 'fog/vsphere/models/compute/server'
 60   Fog::Compute::Vsphere::Server.send(:include, FogExtensions::Vsphere::Server)
 61 
 62   require 'fog/vsphere/models/compute/folder'
 63   Fog::Compute::Vsphere::Folder.send(:include, FogExtensions::Vsphere::Folder)
 64 end

Attempted the same workaround on RedHat Satellite 6.5.3 (based on Foreman 1.2.0) and am still getting the same exact errors as mzima.

It seems this issue appears in various (API, hammer, and webUI) in areas which interact with compute profiles or VMware compute resources in general. As mzima has said, applying the workaround from https://github.com/theforeman/foreman/pull/6803/files seems to change the error to this “ActionView::Template::Error: undefined method `resource_pools’ for nil:NilClass”, but the intermittent issue still persists.

After implementing the workaround to app/models/concerns/fog_extensions.rb:54, and disabling compute resource caching, it seems the issue is resolved, but both were required.

I hope I’ve found solution for this issue: https://github.com/fog/fog-vsphere/pull/267

1 Like

I’m about to update my 4 Foreman servers to 2.1.4 within the next 2 weeks or so. I plan to manually implement your PR and enable the Vsphere caching functionality as well. I will keep you updated how it turns out …

1 Like

Hi,

the fix has been merged and released in fog-vsphere 3.5.0 this will only get released with foreman 2.4, if anyone could confirm it fixed the issue it would be awesome :slight_smile:

Sorry for the late reply.

I’m currently running Foreman 2.1.4. I’ve implemented your PR (backport to tfm-rubygem-fog-vsphere-3.3.1-1) and caching for the Vsphere resources are currently enabled.

I looks like, that I’m still getting very sporadic HTTP 500 Errors when querying /api/compute_resources. But the error message looks less detailed know (maybe I’ve to raise the debug level):

2020-12-23T10:52:09 [I|app|400a1154] Started GET "/api/compute_resources/4?id=4" for 130.197.4.79 at 2020-12-23 10:52:09 +0100
2020-12-23T10:52:09 [I|app|400a1154] Processing by Api::V2::ComputeResourcesController#show as JSON
2020-12-23T10:52:09 [I|app|400a1154]   Parameters: {"id"=>"4", "apiv"=>"v2", "compute_resource"=>{}}
2020-12-23T10:52:09 [I|app|400a1154]   Rendering api/v2/compute_resources/show.json.rabl
2020-12-23T10:52:09 [I|app|2095b854]   Rendering text template
2020-12-23T10:52:09 [I|app|2095b854]   Rendered text template (Duration: 0.0ms | Allocations: 2)
2020-12-23T10:52:09 [I|app|2095b854] Completed 200 OK in 105ms (Views: 0.6ms | ActiveRecord: 18.3ms | Allocations: 34930)
2020-12-23T10:52:09 [I|app|400a1154]   Rendered api/v2/compute_resources/show.json.rabl (Duration: 12.9ms | Allocations: 8283)
2020-12-23T10:52:09 [W|app|400a1154] Action failed
2020-12-23T10:52:09 [I|app|400a1154]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2020-12-23T10:52:09 [I|app|400a1154]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 92)
2020-12-23T10:52:09 [I|app|400a1154] Completed 500 Internal Server Error in 23ms (Views: 0.9ms | ActiveRecord: 2.3ms | Allocations: 11022)