Foreman 1.20.2 / VMware - Sporadic HTTP 500 when querying /api/compute_resources API

Support
#2

Hi, the namespace of the fog-vsphere has been changed, so if you have old data cached, it will fail as you are describing.

But as you are saying it is failing only randomly, it is very weird. Could you share more information with us?
I guess you are running foreman on debian based system, is that correct? If so, could you please share what version is the fog-vsphere in your Gemfile.lock?
Do you have more instances of foreman, or just one?

Could you clean the cache by foreman-rake console and Rails.cache.clear, so we know it was not an old stuff hanging around?

#3

And is it by any chance load-balanced or HA setup?

#4

Hi,

But as you are saying it is failing only randomly, it is very weird. Could you share more information with us?

Yes, this REST query fails kind of randomly. When the query fails I have to resend the REST query (1 to 3 times) before I get a valid result. On the Foreman instance where this issue occurs are 2 VSphere server configured as compute resources (id 4 and 5). It seems like that the problem only appears on resource id 4 and only if caching is active.

I guess you are running foreman on debian based system, is that correct? If so, could you please share what version is the fog-vsphere in your Gemfile.lock ?

Nope, I’m using Foreman 1.20.2 on CentOS 7.6 and the standard fog-vsphere version (tfm-rubygem-fog-vsphere-2.3.0-2.el7.noarch).

Do you have more instances of foreman, or just one?

I’ve in total 4 Foreman instances on different servers (2x 1.19.1 and 2x 1.20.2). All standard installations. No LB or HA setup. I have this issue only on 1 instance (1.20.2)

Could you clean the cache by foreman-rake console and Rails.cache.clear , so we know it was not an old stuff hanging around?

Cleaned the caches … unfortunately no changes.

Cheers,
Martin

#5

Sorry, there was a mistake in my first post. The sporadic error output of the REST query is actually “uninitialized constant Fog::Compute::Vsphere::Cluster”:

$ curl --user *** -k -X GET -H "Content-Type:application/json" -H "Accept:application/json" https://foreman.***.de/api/compute_resources/4 | jq
Enter host password for user '***':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    83    0    83    0     0     26      0 --:--:--  0:00:03 --:--:--    26
{
  "error": {
    "message": "uninitialized constant Fog::Compute::Vsphere::Cluster"
  }
}

I hope this makes more sense now … The rest of my observations remain unchanged.

production.log:

2019-05-15T01:01:50 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 01:01:50 +0200
2019-05-15T01:01:50 [I|app|d8043] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T01:01:50 [I|app|d8043]   Parameters: {"apiv"=>"v2", "id"=>"4", "compute_resource"=>{}}
2019-05-15T01:01:51 [I|app|] Started GET "/notification_recipients" for *** at 2019-05-15 01:01:51 +0200
2019-05-15T01:01:51 [I|app|2dfd1] Processing by NotificationRecipientsController#index as JSON
2019-05-15T01:01:51 [I|app|2dfd1] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|2dfd1] Completed 200 OK in 6ms (Views: 0.2ms | ActiveRecord: 0.6ms)
2019-05-15T01:01:51 [I|app|d8043] Authorized user ***(***)
2019-05-15T01:01:51 [I|app|d8043] Current user set to *** (admin)
2019-05-15T01:01:51 [I|app|d8043]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T01:01:51 [I|app|d8043]   Rendered api/v2/compute_resources/show.json.rabl (22.6ms)
2019-05-15T01:01:51 [W|app|d8043] Action failed
| ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
| /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:158:in `available_clusters'
| /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:627:in `normalize_vm_attrs'
| /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
| /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
| /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl__3693622661913919226_70129623652120'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
| /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
| /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
| /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
| /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T01:01:51 [I|app|d8043]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T01:01:51 [I|app|d8043]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.3ms)
#6

It actually doesn’t :laughing: I gave myself decent time to think about it, and nothing came up.

I really have no idea what could be the cause here, so I can try to guide you through some debugging process, to try figure it out.

You could try to add these at the begging of the method, in which it all goes down.

# app/models/compute_resources

#....

def available_clusters
  require 'fog/vsphere/version'
  Rails.logger.info "current vsphere version is: #{Fog::Vsphere::VERSION}"
  Rails.logger.info "Fog::Compute is #{defined? Fog::Compute}"
  Rails.logger.info "Fog::Compute::Vsphere::Cluster is #{defined? Fog::Compute::Vsphere::Cluster}"
  #...
end

What should give us in the logs answers for:

  • Your code is not reloading fog-vsphere on the fly for another version
  • Fog is not unloaded in between the requests

If everything is all right, I guess we will have to examine the cache for it’s raw value.

#8

Hi,

I’ve added your debug code to the available_clusters method (/usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb)

This is how a successful request looks like:

2019-05-15T18:11:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:11:36 +0200
2019-05-15T18:11:36 [I|app|ef15d] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:11:36 [I|app|ef15d]   Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:11:36 [I|app|ef15d] Current user set to *** (admin)
2019-05-15T18:11:36 [I|app|ef15d]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:11:36 [I|app|ef15d] current vsphere version is: 2.3.0
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute is constant
2019-05-15T18:11:36 [I|app|ef15d] Fog::Compute::Vsphere::Cluster is constant
2019-05-15T18:11:36 [I|app|ef15d]   Rendered api/v2/compute_resources/show.json.rabl (44.2ms)
2019-05-15T18:11:36 [I|app|ef15d] Completed 200 OK in 53ms (Views: 44.1ms | ActiveRecord: 2.4ms)

This was an unsuccessful query:

2019-05-15T18:06:36 [I|app|] Started GET "/api/compute_resources/4" for *** at 2019-05-15 18:06:36 +0200
2019-05-15T18:06:36 [I|app|819fb] Processing by Api::V2::ComputeResourcesController#show as JSON
2019-05-15T18:06:36 [I|app|819fb]   Parameters: {"apiv"=>"v2", "id"=>"4"}
2019-05-15T18:06:36 [I|app|819fb] Current user set to *** (admin)
2019-05-15T18:06:36 [I|app|819fb]   Rendering api/v2/compute_resources/show.json.rabl
2019-05-15T18:06:36 [I|app|819fb] current vsphere version is: 2.3.0
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute is constant
2019-05-15T18:06:36 [I|app|819fb] Fog::Compute::Vsphere::Cluster is 
2019-05-15T18:06:36 [I|app|819fb]   Rendered api/v2/compute_resources/show.json.rabl (24.2ms)
2019-05-15T18:06:36 [W|app|819fb] Action failed
 | ActionView::Template::Error: uninitialized constant Fog::Compute::Vsphere::Cluster
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `const_get'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `inject'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/inflector/methods.rb:281:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:10:in `rescue in load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/marshal.rb:5:in `load'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:804:in `uncompress'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:726:in `value'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:358:in `block in read'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:663:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache.rb:345:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:34:in `read'
 | /usr/share/foreman/app/services/compute_resource_cache.rb:19:in `cache'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:162:in `available_clusters'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:631:in `normalize_vm_attrs'
 | /usr/share/foreman/app/models/compute_attribute.rb:32:in `normalized_vm_attrs'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/helpers.rb:16:in `data_object_attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:140:in `attribute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:114:in `block in compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:113:in `compile_settings'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:53:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:127:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/multi_builder.rb:27:in `to_a'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:88:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:124:in `block (2 levels) in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:123:in `block in merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:119:in `merge_engines_into_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:55:in `block in to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:248:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/builder.rb:49:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:86:in `to_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:99:in `to_dumpable'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:108:in `to_json'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:48:in `block in render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:380:in `cache_results'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rabl-0.13.1/lib/rabl/engine.rb:47:in `render'
 | /usr/share/foreman/config/initializers/rabl_init.rb:49:in `render'
 | /usr/share/foreman/app/views/api/v2/compute_resources/show.json.rabl:3:in `_app_views_api_v__compute_resources_show_json_rabl___541234303595413005_70067119553160'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
 | /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
 | /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-05-15T18:06:36 [I|app|819fb]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-05-15T18:06:36 [I|app|819fb]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.4ms)
2019-05-15T18:06:36 [I|app|819fb] Completed 500 Internal Server Error in 35ms (Views: 1.1ms | ActiveRecord: 2.3ms)

Cheers,
Martin

#9

Yesterday I’ve updated another Foreman server to 1.20.2 which is now showing the same issue (sporadic HTTP 500 / “uninitialized constant Fog::Compute::Vsphere::Cluster”. This Foreman Server uses the same VSphere Server to create hosts as the other one.

#10

Can you please try to do a “Refresh Cache” on the “Infrastructure -> Compute Resources -> Your Vmware Provider” page?

Does it work afterwards? If this is the case, does it fail again?

#11

In my experience it works after refreshing the cache, but it does start to fail again after a short while. The only workaround I have right now is to disable the compute resource cache completely, which results in very slow host edit pages.

#12

I’m having similar problems on 1.21.3, but I believe it happened for me on 1.20 as well.

Typically it happens when trying to edit a host in the GUI. Now and then we get the error message:

Oops, we're sorry but something went wrong uninitialized constant Fog::Compute::Vsphere::Network Did you mean? Fog::Network

We then just try again and then it works at the second or third attempt. I just did a ‘refresh cache’ on my vmware compute resource, but that didn’t help.

We’re on RHEL7 with standard rpm installation, standalone installation (no HA, no LB)
tfm-rubygem-fog-vsphere-2.5.0-1.el7.noarch

#13

I believe it is something about rails autoloading vs passanger and fog nasty namespaces. I wonder though how comes we are hitting those now out of the blue :thinking:
I had blew too much time on trying to fix that and there is recently introduced simple patch by, which should have fix that:


That is not rooting out the evil, but it is solving the issue, could you guys test it, if it works for you?

#14

YES, that commit seems to have fixed the issue in my environment.
Good enough for me, thank you!

#15

Hi,

I’m still getting sporadic HTTP 500 errors after applying the patch (same REST query, same error frequency). However the error message is now a different one:

ActionView::Template::Error: undefined method `resource_pools’ for nil:NilClass

production.log snippet:

production_log_snippet.log (20.5 KB)

Greetings,
Martin

#16

I also get another, similar error messages in production.log, but not as frequent. I got these before the patch as well, but I don’t know exactly in what situation since it to my knowledge hasnt affected any users.

Error messages like:

2019-05-28T05:50:23 [W|app|b2706f9c] Action failed
 | NameError: uninitialized constant Fog::Compute::Vsphere::StoragePod
 | Did you mean?  Fog::Storage
#17

Could you guys please try to add

# app/models/concerns/fog_extensions.rb:54
require 'fog/vsphere/compute'

and try again?

#18

I already have that line on 1.21

#19

the line is already included (1.20.2):

 54 if Foreman::Model::Vmware.available?
 55   require 'fog/vsphere'
 56   require 'fog/vsphere/models/compute/cluster'
 57   require 'fog/vsphere/models/compute/network'
 58   require 'fog/vsphere/compute'
 59   require 'fog/vsphere/models/compute/server'
 60   Fog::Compute::Vsphere::Server.send(:include, FogExtensions::Vsphere::Server)
 61 
 62   require 'fog/vsphere/models/compute/folder'
 63   Fog::Compute::Vsphere::Folder.send(:include, FogExtensions::Vsphere::Folder)
 64 end
#20

Attempted the same workaround on RedHat Satellite 6.5.3 (based on Foreman 1.2.0) and am still getting the same exact errors as mzima.

It seems this issue appears in various (API, hammer, and webUI) in areas which interact with compute profiles or VMware compute resources in general. As mzima has said, applying the workaround from https://github.com/theforeman/foreman/pull/6803/files seems to change the error to this “ActionView::Template::Error: undefined method `resource_pools’ for nil:NilClass”, but the intermittent issue still persists.

#21

After implementing the workaround to app/models/concerns/fog_extensions.rb:54, and disabling compute resource caching, it seems the issue is resolved, but both were required.

#22

I hope I’ve found solution for this issue: https://github.com/fog/fog-vsphere/pull/267

1 Like