Foreman 1.20.2 / VMware - Sporadic HTTP 500 when querying /api/compute_resources API


#1

Problem:

We are getting sporadic HTTP 500 Errors (see below) when executing the following REST operation on a VMware compute resource since updating Foreman from 1.19.1 to 1.20.2:

/api/compute_resources/4?id=4

The weird thing is, that we are getting the expected result after re-sending the query. It seems like, that the problem only occurs when enabling caching for the VMware compute resource.

Has anyone an idea what might cause this behavior?

Foreman and Proxy versions:

  • Foreman 1.20.2
  • Foreman Proxy 1.20.2
  • Foreman Plugins 1.20.2

Vsphere version:

  • 6.5 Update 2e

Error output:

{"error": {"message": "Fog::Compute::Vsphere::NotFound"}}

Logfile /var/log/foreman/production.log

2019-05-14T07:26:30 [I|app|] Started GET "/api/compute_resources/4?id=4" for 130.*** at 2019-05-14 07:26:30 +0200
2019-05-14T07:26:30 [I|app|c40f1] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-14T07:26:30 [I|app|c40f1]   Parameters: {"id"=>"4", "apiv"=>"v2", "compute_resource"=>{}}
2019-05-14T07:26:30 [I|app|c40f1] Current user set to *** (admin)
2019-05-14T07:26:30 [I|app|c40f1]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-14T07:26:30 [I|app|c40f1]   Rendered api/v2/compute_resources/show.json.rabl (42.4ms)
2019-05-14T07:26:30 [W|app|c40f1] Action failed
 | ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:158:in `available_clusters'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:627:in `normalize_vm_attrs'
 | /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl___4452855598344156419_70339316387680'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
 | /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
 | /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
 | /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-14T07:26:30 [I|app|c40f1]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-14T07:26:30 [I|app|c40f1]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.3ms)
2019-05-14T07:26:30 [I|app|c40f1] Completed 500 Internal Server Error in 57ms (Views: 1.0ms | ActiveRecord: 3.9ms)

#2

Hi, the namespace of the fog-vsphere has been changed, so if you have old data cached, it will fail as you are describing.

But as you are saying it is failing only randomly, it is very weird. Could you share more information with us?
I guess you are running foreman on debian based system, is that correct? If so, could you please share what version is the fog-vsphere in your Gemfile.lock?
Do you have more instances of foreman, or just one?

Could you clean the cache by foreman-rake console and Rails.cache.clear, so we know it was not an old stuff hanging around?


#3

And is it by any chance load-balanced or HA setup?


#4

Hi,

But as you are saying it is failing only randomly, it is very weird. Could you share more information with us?

Yes, this REST query fails kind of randomly. When the query fails I have to resend the REST query (1 to 3 times) before I get a valid result. On the Foreman instance where this issue occurs are 2 VSphere server configured as compute resources (id 4 and 5). It seems like that the problem only appears on resource id 4 and only if caching is active.

I guess you are running foreman on debian based system, is that correct? If so, could you please share what version is the fog-vsphere in your Gemfile.lock ?

Nope, I’m using Foreman 1.20.2 on CentOS 7.6 and the standard fog-vsphere version (tfm-rubygem-fog-vsphere-2.3.0-2.el7.noarch).

Do you have more instances of foreman, or just one?

I’ve in total 4 Foreman instances on different servers (2x 1.19.1 and 2x 1.20.2). All standard installations. No LB or HA setup. I have this issue only on 1 instance (1.20.2)

Could you clean the cache by foreman-rake console and Rails.cache.clear , so we know it was not an old stuff hanging around?

Cleaned the caches … unfortunately no changes.

Cheers,
Martin


#5

Sorry, there was a mistake in my first post. The sporadic error output of the REST query is actually “uninitialized constant Fog::Compute::Vsphere::Cluster”:

$ curl --user *** -k -X GET -H "Content-Type:application/json" -H "Accept:application/json" https://foreman.***.de/api/compute_resources/4 | jq
Enter host password for user '***':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    83    0    83    0     0     26      0 --:--:--  0:00:03 --:--:--    26
{
  "error": {
    "message": "uninitialized constant Fog::Compute::Vsphere::Cluster"
  }
}

I hope this makes more sense now … The rest of my observations remain unchanged.

production.log:

2019-05-15T01:01:50 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 01:01:50 +0200
2019-05-15T01:01:50 [I|app|d8043] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T01:01:50 [I|app|d8043]   Parameters: {"apiv"=>"v2", "id"=>"4", "compute_resource"=>{}}
2019-05-15T01:01:51 [I|app|] Started GET "/notification_recipients" for *** at 2019-05-15 01:01:51 +0200
2019-05-15T01:01:51 [I|app|2dfd1] Processing by NotificationRecipientsController#index as JSON
2019-05-15T01:01:51 [I|app|2dfd1] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|2dfd1] Completed 200 OK in 6ms (Views: 0.2ms | ActiveRecord: 0.6ms)
2019-05-15T01:01:51 [I|app|d8043] Authorized user ***(***)
2019-05-15T01:01:51 [I|app|d8043] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|d8043]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T01:01:51 [I|app|d8043]   Rendered api/v2/compute_resources/show.json.rabl (22.6ms)
2019-05-15T01:01:51 [W|app|d8043] Action failed
| ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:158:in `available_clusters'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:627:in `normalize_vm_attrs'
| /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl__3693622661913919226_70129623652120'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
| /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
| /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
| /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
| /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T01:01:51 [I|app|d8043]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T01:01:51 [I|app|d8043]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.3ms)

#6

It actually doesn’t :laughing: I gave myself decent time to think about it, and nothing came up.

I really have no idea what could be the cause here, so I can try to guide you through some debugging process, to try figure it out.

You could try to add these at the begging of the method, in which it all goes down.

# app/models/compute_resources

#....

def available_clusters
  require 'fog/vsphere/version'
  Rails.logger.info "current vsphere version is: #{Fog::Vsphere::VERSION}"
  Rails.logger.info "Fog::Compute is #{defined? Fog::Compute}"
  Rails.logger.info "Fog::Compute::Vsphere::Cluster is #{defined? Fog::Compute::Vsphere::Cluster}"
  #...
end

What should give us in the logs answers for:

  • Your code is not reloading fog-vsphere on the fly for another version
  • Fog is not unloaded in between the requests

If everything is all right, I guess we will have to examine the cache for it’s raw value.


#8

Hi,

I’ve added your debug code to the available_clusters method (/usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb)

This is how a successful request looks like:

2019-05-15T18:11:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:11:36 +0200
2019-05-15T18:11:36 [I|app|ef15d] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:11:36 [I|app|ef15d]   Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:11:36 [I|app|ef15d] Current user set to *** (admin)
2019-05-15T18:11:36 [I|app|ef15d]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:11:36 [I|app|ef15d] current vsphere version is: 2.3.0
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute is constant
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute::Vsphere::Cluster is constant
2019-05-15T18:11:36 [I|app|ef15d]   Rendered api/v2/compute_resources/show.json.rabl (44.2ms)
2019-05-15T18:11:36 [I|app|ef15d] Completed 200 OK in 53ms (Views: 44.1ms | ActiveRecord: 2.4ms)

This was an unsuccessful query:

2019-05-15T18:06:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:06:36 +0200
2019-05-15T18:06:36 [I|app|819fb] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:06:36 [I|app|819fb]   Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:06:36 [I|app|819fb] Current user set to *** (admin)
2019-05-15T18:06:36 [I|app|819fb]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:06:36 [I|app|819fb] current vsphere version is: 2.3.0
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute is constant
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute::Vsphere::Cluster is 
2019-05-15T18:06:36 [I|app|819fb]   Rendered api/v2/compute_resources/show.json.rabl (24.2ms)
2019-05-15T18:06:36 [W|app|819fb] Action failed
 | ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:162:in `available_clusters'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:631:in `normalize_vm_attrs'
 | /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl___541234303595413005_70067119553160'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
 | /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
 | /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T18:06:36 [I|app|819fb]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T18:06:36 [I|app|819fb]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.4ms)
2019-05-15T18:06:36 [I|app|819fb] Completed 500 Internal Server Error in 35ms (Views: 1.1ms | ActiveRecord: 2.3ms)

Cheers,
Martin


#9

Yesterday I’ve updated another Foreman server to 1.20.2 which is now showing the same issue (sporadic HTTP 500 / “uninitialized constant Fog::Compute::Vsphere::Cluster”. This Foreman Server uses the same VSphere Server to create hosts as the other one.