mzima
May 14, 2019, 1:24pm
1
Problem:
We are getting sporadic HTTP 500 Errors (see below) when executing the following REST operation on a VMware compute resource since updating Foreman from 1.19.1 to 1.20.2 :
/api/compute_resources/4?id=4
The weird thing is, that we are getting the expected result after re-sending the query. It seems like, that the problem only occurs when enabling caching for the VMware compute resource.
Has anyone an idea what might cause this behavior?
Foreman and Proxy versions:
Foreman 1.20.2
Foreman Proxy 1.20.2
Foreman Plugins 1.20.2
Vsphere version:
Error output:
{"error": {"message": "Fog::Compute::Vsphere::NotFound"}}
Logfile /var/log/foreman/production.log
2019-05-14T07:26:30 [I|app|] Started GET "/api/compute_resources/4?id=4" for 130.*** at 2019-05-14 07:26:30 +0200
2019-05-14T07:26:30 [I|app|c40f1] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-14T07:26:30 [I|app|c40f1] Parameters: {"id"=>"4", "apiv"=>"v2", "compute_resource"=>{}}
2019-05-14T07:26:30 [I|app|c40f1] Current user set to *** (admin)
2019-05-14T07:26:30 [I|app|c40f1] Rendering api/v2/compute_resources/show.json.rabl
2019-05-14T07:26:30 [I|app|c40f1] Rendered api/v2/compute_resources/show.json.rabl (42.4ms)
2019-05-14T07:26:30 [W|app|c40f1] Action failed
| ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:158:in `available_clusters'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:627:in `normalize_vm_attrs'
| /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl___4452855598344156419_70339316387680'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
| /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
| /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
| /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
| /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-14T07:26:30 [I|app|c40f1] Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-14T07:26:30 [I|app|c40f1] Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.3ms)
2019-05-14T07:26:30 [I|app|c40f1] Completed 500 Internal Server Error in 57ms (Views: 1.0ms | ActiveRecord: 3.9ms)
Hi, the namespace of the fog-vsphere has been changed, so if you have old data cached, it will fail as you are describing.
But as you are saying it is failing only randomly, it is very weird. Could you share more information with us?fog-vsphere in your Gemfile.lock?
Could you clean the cache by foreman-rake console and Rails.cache.clear, so we know it was not an old stuff hanging around?
And is it by any chance load-balanced or HA setup?
mzima
May 14, 2019, 4:54pm
4
Hi,
But as you are saying it is failing only randomly, it is very weird. Could you share more information with us?
Yes, this REST query fails kind of randomly. When the query fails I have to resend the REST query (1 to 3 times) before I get a valid result. On the Foreman instance where this issue occurs are 2 VSphere server configured as compute resources (id 4 and 5). It seems like that the problem only appears on resource id 4 and only if caching is active.
I guess you are running foreman on debian based system, is that correct? If so, could you please share what version is the fog-vsphere in your Gemfile.lock ?
Nope, Iām using Foreman 1.20.2 on CentOS 7.6 and the standard fog-vsphere version (tfm-rubygem-fog-vsphere-2.3.0-2.el7.noarch).
Do you have more instances of foreman, or just one?
Iāve in total 4 Foreman instances on different servers (2x 1.19.1 and 2x 1.20.2). All standard installations. No LB or HA setup. I have this issue only on 1 instance (1.20.2)
Could you clean the cache by foreman-rake console and Rails.cache.clear , so we know it was not an old stuff hanging around?
Cleaned the caches ā¦ unfortunately no changes.
Cheers,
mzima
May 14, 2019, 11:13pm
5
Sorry, there was a mistake in my first post. The sporadic error output of the REST query is actually āuninitialized constant Fog::Compute::Vsphere::Cluster ā:
$ curl --user *** -k -X GET -H "Content-Type:application/json" -H "Accept:application/json" https://foreman.***.de/api/compute_resources/4 | jq
Enter host password for user '***':
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 83 0 83 0 0 26 0 --:--:-- 0:00:03 --:--:-- 26
{
"error": {
"message": "uninitialized constant Fog::Compute::Vsphere::Cluster"
}
}
I hope this makes more sense now ā¦ The rest of my observations remain unchanged.
production.log:
2019-05-15T01:01:50 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 01:01:50 +0200
2019-05-15T01:01:50 [I|app|d8043] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T01:01:50 [I|app|d8043] Parameters: {"apiv"=>"v2", "id"=>"4", "compute_resource"=>{}}
2019-05-15T01:01:51 [I|app|] Started GET "/notification_recipients" for *** at 2019-05-15 01:01:51 +0200
2019-05-15T01:01:51 [I|app|2dfd1] Processing by NotificationRecipientsController#index as JSON
2019-05-15T01:01:51 [I|app|2dfd1] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|2dfd1] Completed 200 OK in 6ms (Views: 0.2ms | ActiveRecord: 0.6ms)
2019-05-15T01:01:51 [I|app|d8043] Authorized user ***(***)
2019-05-15T01:01:51 [I|app|d8043] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|d8043] Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T01:01:51 [I|app|d8043] Rendered api/v2/compute_resources/show.json.rabl (22.6ms)
2019-05-15T01:01:51 [W|app|d8043] Action failed
| ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:158:in `available_clusters'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:627:in `normalize_vm_attrs'
| /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl__3693622661913919226_70129623652120'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
| /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
| /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
| /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
| /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T01:01:51 [I|app|d8043] Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T01:01:51 [I|app|d8043] Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.3ms)
It actually doesnāt
I really have no idea what could be the cause here, so I can try to guide you through some debugging process, to try figure it out.
You could try to add these at the begging of the method, in which it all goes down.
# app/models/compute_resources
#....
def available_clusters
require 'fog/vsphere/version'
Rails.logger.info "current vsphere version is: #{Fog::Vsphere::VERSION}"
Rails.logger.info "Fog::Compute is #{defined? Fog::Compute}"
Rails.logger.info "Fog::Compute::Vsphere::Cluster is #{defined? Fog::Compute::Vsphere::Cluster}"
#...
end
What should give us in the logs answers for:
Your code is not reloading fog-vsphere on the fly for another version
Fog is not unloaded in between the requests
If everything is all right, I guess we will have to examine the cache for itās raw value.
mzima
May 15, 2019, 4:30pm
8
Hi,
Iāve added your debug code to the available_clusters method (/usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb)
This is how a successful request looks like:
2019-05-15T18:11:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:11:36 +0200
2019-05-15T18:11:36 [I|app|ef15d] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:11:36 [I|app|ef15d] Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:11:36 [I|app|ef15d] Current user set to *** (admin)
2019-05-15T18:11:36 [I|app|ef15d] Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:11:36 [I|app|ef15d] current vsphere version is: 2.3.0
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute is constant
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute::Vsphere::Cluster is constant
2019-05-15T18:11:36 [I|app|ef15d] Rendered api/v2/compute_resources/show.json.rabl (44.2ms)
2019-05-15T18:11:36 [I|app|ef15d] Completed 200 OK in 53ms (Views: 44.1ms | ActiveRecord: 2.4ms)
This was an unsuccessful query:
2019-05-15T18:06:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:06:36 +0200
2019-05-15T18:06:36 [I|app|819fb] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:06:36 [I|app|819fb] Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:06:36 [I|app|819fb] Current user set to *** (admin)
2019-05-15T18:06:36 [I|app|819fb] Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:06:36 [I|app|819fb] current vsphere version is: 2.3.0
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute is constant
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute::Vsphere::Cluster is
2019-05-15T18:06:36 [I|app|819fb] Rendered api/v2/compute_resources/show.json.rabl (24.2ms)
2019-05-15T18:06:36 [W|app|819fb] Action failed
| ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:162:in `available_clusters'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:631:in `normalize_vm_attrs'
| /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl___541234303595413005_70067119553160'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
| /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
| /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
| /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
| /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T18:06:36 [I|app|819fb] Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T18:06:36 [I|app|819fb] Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.4ms)
2019-05-15T18:06:36 [I|app|819fb] Completed 500 Internal Server Error in 35ms (Views: 1.1ms | ActiveRecord: 2.3ms)
Cheers,
mzima
May 17, 2019, 8:06am
9
Yesterday Iāve updated another Foreman server to 1.20.2 which is now showing the same issue (sporadic HTTP 500 / āuninitialized constant Fog::Compute::Vsphere::Clusterā. This Foreman Server uses the same VSphere Server to create hosts as the other one.
Can you please try to do a āRefresh Cacheā on the āInfrastructure -> Compute Resources -> Your Vmware Providerā page?
Does it work afterwards? If this is the case, does it fail again?
mzima
May 25, 2019, 11:02am
11
In my experience it works after refreshing the cache, but it does start to fail again after a short while. The only workaround I have right now is to disable the compute resource cache completely, which results in very slow host edit pages.
Iām having similar problems on 1.21.3, but I believe it happened for me on 1.20 as well.
Typically it happens when trying to edit a host in the GUI. Now and then we get the error message:
Oops, we're sorry but something went wrong uninitialized constant Fog::Compute::Vsphere::Network Did you mean? Fog::Network
We then just try again and then it works at the second or third attempt. I just did a ārefresh cacheā on my vmware compute resource, but that didnāt help.
Weāre on RHEL7 with standard rpm installation, standalone installation (no HA, no LB)
I believe it is something about rails autoloading vs passanger and fog nasty namespaces. I wonder though how comes we are hitting those now out of the blue
theforeman:develop ā ATIX-AG:fix_vsphere_network
opened 01:35PM - 27 May 19 UTC
Due to a race condition, sometimes the the failure 'uninitialized constant Fog::ā¦ Compute::Vsphere::Network' happens when using the compute-resource networks API.
<!--
Simple description of what is fixed/introduced.
Prerequisites for testing:
* VMware cluster
* Two smart proxies
Tests needed (anticipated impacts):
* Hosts list - mainly power column
* Power status on host page
-->
<!---
Thank you for contributing to The Foreman, please read the
[following guide](https://www.theforeman.org/contribute.html), in short:
* [Create an issue](https://projects.theforeman.org/projects/foreman/issues)
* Reference the issue via `Fixes #1234` in the commit message
* Prefer present-tense, imperative-style commit messages
* Mark all strings for translation, see [Translating section in the guide]
* Suggest prerequisites for testing and testing scenarios following example above.
(https://projects.theforeman.org/projects/foreman/wiki/Translating)
* Prepend `[WIP]` for work in progress to prevent bots from triggering actions
* Be patient, we will do our best to take a look as soon as we can
* Explain the purpose of the PR, attach screenshots if applicable
* Be nice and respectful
We are running bots that will poke you if you miss an item from the list :-)
-->
That is not rooting out the evil, but it is solving the issue, could you guys test it, if it works for you?
YES, that commit seems to have fixed the issue in my environment.
mzima
May 28, 2019, 7:51am
15
Hi,
Iām still getting sporadic HTTP 500 errors after applying the patch (same REST query, same error frequency). However the error message is now a different one:
ActionView::Template::Error: undefined method `resource_poolsā for nil:NilClass
production.log snippet:
production_log_snippet.log (20.5 KB)
Greetings,
I also get another, similar error messages in production.log, but not as frequent. I got these before the patch as well, but I donāt know exactly in what situation since it to my knowledge hasnt affected any users.
Error messages like:
2019-05-28T05:50:23 [W|app|b2706f9c] Action failed
| NameError: uninitialized constant Fog::Compute::Vsphere::StoragePod
| Did you mean? Fog::Storage
Could you guys please try to add
# app/models/concerns/fog_extensions.rb:54
require 'fog/vsphere/compute'
and try again?
I already have that line on 1.21
mzima
May 28, 2019, 10:02am
19
the line is already included (1.20.2):
54 if Foreman::Model::Vmware.available?
55 require 'fog/vsphere'
56 require 'fog/vsphere/models/compute/cluster'
57 require 'fog/vsphere/models/compute/network'
58 require 'fog/vsphere/compute'
59 require 'fog/vsphere/models/compute/server'
60 Fog::Compute::Vsphere::Server.send(:include, FogExtensions::Vsphere::Server)
61
62 require 'fog/vsphere/models/compute/folder'
63 Fog::Compute::Vsphere::Folder.send(:include, FogExtensions::Vsphere::Folder)
64 end
Attempted the same workaround on RedHat Satellite 6.5.3 (based on Foreman 1.2.0) and am still getting the same exact errors as mzima.
It seems this issue appears in various (API, hammer, and webUI) in areas which interact with compute profiles or VMware compute resources in general. As mzima has said, applying the workaround from https://github.com/theforeman/foreman/pull/6803/files seems to change the error to this āActionView::Template::Error: undefined method `resource_poolsā for nil:NilClassā, but the intermittent issue still persists.
After implementing the workaround to app/models/concerns/fog_extensions.rb:54, and disabling compute resource caching, it seems the issue is resolved, but both were required.
I gave myself decent time to think about it, and nothing came up.