Failed to create a compute CXX01 (VMware) instance serverxyz1.domain.com: undefined method `_ref' for nil:NilClass

Problem:
When trying to create a new Virtual machine (Vmware) it failes with Error:
Failed to create a compute CXX01 (VMware) instance serverxyz1.domain.com: undefined method `_ref’ for nil:NilClass
As the provisining was already working, there must be an issue either in the data foreman is sending, or on vmware side, but i’m not able to find it.

Expected outcome:
Creation of a virtual machine

Foreman version:
3.2.1

Other relevant data:
Here the output from production.log.
Have checked that with our vcenter colleagues. They only can see, that the user is logged in, but no error or anything else. They have granted the user admin permissions too (for tests) but that does not change anything.

2023-01-02T12:53:58 [I|app|4b2cfc41] Started POST “/api/hosts” for 10.10.66.174 at 2023-01-02 12:53:58 +0100
2023-01-02T12:53:58 [I|app|4b2cfc41] Processing by Api::V2::HostsController#create as JSON
2023-01-02T12:53:58 [I|app|4b2cfc41] Parameters: {“host”=>{“name”=>“serverxyz1”, “hostgroup_id”=>32, “compute_resource_id”=>11, “managed”=>true, “type”=>“Host::Managed”, “provision_method”=>“build”, “build”=>true, “is_owned_by”=>“osstool”, “enabled”=>true, “overwrite”=>false, “root_pass”=>"[FILTERED]", “puppet_attributes”=>{“environment_id”=>4}, “host_parameters_attributes”=>{“0”=>{“name”=>“provisioned_by”, “value”=>"[FILTERED]", “parameter_type”=>“string”}, “1”=>{“name”=>“ci_environment”, “value”=>"[FILTERED]", “parameter_type”=>“string”}, “2”=>{“name”=>“ci_label”, “value”=>"[FILTERED]", “parameter_type”=>“string”}, “3”=>{“name”=>“ciid”, “value”=>"[FILTERED]", “parameter_type”=>“string”}}, “interfaces_attributes”=>{“0”=>{"_destroy"=>false, “type”=>“Nic::Managed”, “identifier”=>“eth0”, “name”=>“serverxyz1”, “domain_id”=>1, “subnet_id”=>204, “ip”=>“10.191.139.134”, “managed”=>true, “primary”=>true, “provision”=>true, “execution”=>true, “virtual”=>false, “compute_attributes”=>{“type”=>“VirtualVmxnet3”, “network”=>“010_191_139_128_VSR”}}}, “compute_attributes”=>{“cpus”=>2, “corespersocket”=>2, “memory_mb”=>4096, “firmware”=>“efi”, “cluster”=>“LIN-NONPROD”, “resource_pool”=>“Resources”, “path”=>"/Datencenter/DC-CXX01/vm/EXX/Autoprovisioning", “guest_id”=>“rhel8_64Guest”, “hardware_version”=>“Default”, “memoryHotAddEnabled”=>true, “cpuHotAddEnabled”=>true, “add_cdrom”=>false, “boot_order”=>[“disk”, “network”], “start”=>“1”, “scsi_controllers”=>[{“type”=>“ParaVirtualSCSIController”, “key”=>“1000”}], “volumes_attributes”=>{“0”=>{“controllerKey”=>“1000”, “datastore”=>“LIN_CCP_FCP010_DEV”, “eager_zero”=>false, “name”=>“OS Disk”, “size_gb”=>30, “thin”=>true}}}}, “apiv”=>“v2”}
2023-01-02T12:53:58 [I|app|4b2cfc41] Authorized user api_provisioning(API Prov)
2023-01-02T12:53:59 [W|app|4b2cfc41] Scoped order is ignored, it’s forced to be batch order.
2023-01-02T12:53:59 [I|app|4b2cfc41] Adding Compute instance for serverxyz1.domain.com
2023-01-02T12:54:01 [W|app|4b2cfc41] Failed to create a compute CXX01 (VMware) instance serverxyz1.domain.com: undefined method _ref' for nil:NilClass 4b2cfc41 | 2023-01-02T12:54:01 [I|app|4b2cfc41] Backtrace for 'Failed to create a compute CXX01 (VMware) instance serverxyz1.domain.com: undefined method _ref’ for nil:NilClass
4b2cfc41 | ’ error (NoMethodError): undefined method _ref' for nil:NilClass 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/fog-vsphere-3.5.0/lib/fog/vsphere/requests/compute/list_networks.rb:30:in block in list_networks’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/fog-vsphere-3.5.0/lib/fog/vsphere/requests/compute/list_networks.rb:24:in map' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/fog-vsphere-3.5.0/lib/fog/vsphere/requests/compute/list_networks.rb:24:in list_networks’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/fog-vsphere-3.5.0/lib/fog/vsphere/models/compute/networks.rb:10:in all' 4b2cfc41 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:152:in block in networks’
4b2cfc41 | /usr/share/foreman/app/services/compute_resource_cache.rb:68:in instance_eval' 4b2cfc41 | /usr/share/foreman/app/services/compute_resource_cache.rb:68:in get_uncached_value’
4b2cfc41 | /usr/share/foreman/app/services/compute_resource_cache.rb:22:in cache' 4b2cfc41 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:151:in networks’
4b2cfc41 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:471:in parse_networks' 4b2cfc41 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:488:in create_vm’
4b2cfc41 | /usr/share/foreman/app/models/concerns/orchestration/compute.rb:107:in setCompute' 4b2cfc41 | /usr/share/foreman/app/models/concerns/orchestration.rb:227:in execute’
4b2cfc41 | /usr/share/foreman/app/models/concerns/orchestration.rb:152:in block in process' 4b2cfc41 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in each’
4b2cfc41 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in process' 4b2cfc41 | /usr/share/foreman/app/models/concerns/orchestration.rb:44:in around_save_orchestration’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in run_callbacks’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:825:in _run_save_callbacks' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:327:in create_or_update’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/timestamp.rb:128:in create_or_update' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:470:in save’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:47:in save' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:314:in block in save’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in transaction' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:366:in with_transaction_returning_status’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:314:in save' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:44:in save’
4b2cfc41 | /usr/share/foreman/app/models/concerns/foreman/sti.rb:26:in save' 4b2cfc41 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:147:in create’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in send_action' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in process_action’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in process_action' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in block in process_action’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in block in run_callbacks' 4b2cfc41 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 4b2cfc41 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in clear_thread’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 4b2cfc41 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in run_callbacks’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in process_action' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in process_action’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in block in process_action' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in block in instrument’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in instrument' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in instrument’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in process_action' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in process_action’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in process_action' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in process’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in process' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in dispatch’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in dispatch' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in dispatch’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in serve' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:18:in block in class:Constraints
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:48:in serve' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in block in serve’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in each' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in serve’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call' 4b2cfc41 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in call’
4b2cfc41 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in call’
4b2cfc41 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in call’
4b2cfc41 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in context’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in block in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in run_callbacks’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in call_app' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call' 4b2cfc41 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in public_send’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in method_missing' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/configuration.rb:249:in call' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/request.rb:77:in block in handle_request’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/thread_pool.rb:340:in with_force_shutdown' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/request.rb:76:in handle_request’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/server.rb:447:in process_client' 4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/thread_pool.rb:147:in block in spawn_thread’
4b2cfc41 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

2023-01-02T12:54:01 [W|app|4b2cfc41] Rolling back due to a problem: [#<Orchestration::Task:0x000000000fe4e7d8 @name=“Set up compute instance serverxyz1.domain.com”, @id=“Set up compute instance serverxyz1.domain.com”, @status=“failed”, @priority=3, @action=[#<Host::Managed id: nil, name: “serverxyz1.domain.com”, last_compile: nil, last_report: nil, updated_at: nil, created_at: nil, root_pass: nil, architecture_id: 1, operatingsystem_id: 159, ptable_id: 86, medium_id: 15, build: true, comment: nil, disk: nil, installed_at: nil, model_id: nil, hostgroup_id: 32, owner_id: 38, owner_type: “User”, enabled: true, puppet_ca_proxy_id: 3, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: 11, puppet_proxy_id: 3, certname: nil, image_id: nil, organization_id: 2, location_id: 1, type: “Host::Managed”, compute_profile_id: nil, otp: nil, realm_id: nil, provision_method: “build”, grub_pass: nil, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: nil, initiated_at: nil, build_errors: nil>, :setCompute], @created=1672660439.3466935, @timestamp=2023-01-02 11:54:01.023793981 UTC>]

2023-01-02T12:54:01 [I|app|4b2cfc41] Processed 1 tasks from queue ‘Host::Managed Main’, completed 0/11
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Set up compute instance serverxyz1.domain.comfailed
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Query instance details for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Create DHCP Settings for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Create IPv4 DNS record for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Create Reverse IPv4 DNS record for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Deploy TFTP PXELinux config for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Deploy TFTP PXEGrub2 config for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Deploy TFTP PXEGrub config for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Deploy TFTP iPXE config for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Fetch TFTP boot files for serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Task ‘Power up compute instance serverxyz1.domain.comcanceled
2023-01-02T12:54:01 [E|app|4b2cfc41] Unprocessable entity Host::Managed (id: new):
4b2cfc41 | Failed to create a compute CXX01 (VMware) instance serverxyz1.domain.com: undefined method `_ref’ for nil:NilClass
4b2cfc41 |
4b2cfc41 |
2023-01-02T12:54:01 [I|app|4b2cfc41] Rendering api/v2/errors/unprocessable_entity.json.rabl within api/v2/layouts/error_layout
2023-01-02T12:54:01 [I|app|4b2cfc41] Rendered api/v2/errors/unprocessable_entity.json.rabl within api/v2/layouts/error_layout (Duration: 4.7ms | Allocations: 5287)
2023-01-02T12:54:01 [I|app|4b2cfc41] Completed 422 Unprocessable Entity in 2151ms (Views: 9.3ms | ActiveRecord: 81.5ms | Allocations: 2379195)

Is there a way to get better log lines instead of this exception?
Any ideas what i can try?

Thanks in advanced.
BR
Flo

Update on this.
A colleague of me has made some changes in the Fog Library and added some improvements to response messages and log entries. So it turned out, that our user was missing some permissions on vcenter side. Even with admin permissions it doesn’t work. It seems that Vcenter permissions system is a little bit confusing.

So problem is solved.

2 Likes

Hi @FSP

Would it be possible for you to submit a pr to fog-vsphere with the improvments?

Hello @FSP

I know it’s quite late to ask this but is there any way you remember and can share, that how your colleague did the debugging in the past?

I am interested in the code changes made to improve the response message and log entries

– Sayan

Hi,
sorry for the late answer.
In fact i cannot tell you the code update he did. We updated the foreman version some weeks after this change, and in the new version the error was not present anymore.
Sorry
Flo