I finally found something in the foreman logs, after reenabling the debug output.
Here is the log entry i found:
2024-11-28T14:14:18 [W|app|85972389] Failed to create a compute vsphere (VMware) instance foreman-bug.xxx.xxx: InvalidProperty:
85972389 |
2024-11-28T14:14:18 [I|app|85972389] Backtrace for 'Failed to create a compute vsphere (VMware) instance foreman-bug.xxx.xxx: InvalidProperty:
85972389 | ' error (RbVmomi::Fault): InvalidProperty:
85972389 | /usr/share/gems/gems/rbvmomi2-3.8.0/lib/rbvmomi/connection.rb:64:in `parse_response'
85972389 | /usr/share/gems/gems/rbvmomi2-3.8.0/lib/rbvmomi/connection.rb:93:in `call'
85972389 | /usr/share/gems/gems/rbvmomi2-3.8.0/lib/rbvmomi/basic_types.rb:256:in `_call'
85972389 | /usr/share/gems/gems/rbvmomi2-3.8.0/lib/rbvmomi/basic_types.rb:77:in `block (2 levels) in init'
85972389 | /usr/share/gems/gems/rbvmomi2-3.8.0/lib/rbvmomi/vim/ManagedObject.rb:42:in `collect!'
85972389 | /usr/share/gems/gems/fog-vsphere-3.7.0/lib/fog/vsphere/compute.rb:195:in `convert_vm_mob_ref_to_attr_hash'
85972389 | /usr/share/gems/gems/fog-vsphere-3.7.0/lib/fog/vsphere/requests/compute/get_virtual_machine.rb:7:in `get_virtual_machine'
85972389 | /usr/share/gems/gems/fog-vsphere-3.7.0/lib/fog/vsphere/models/compute/servers.rb:29:in `get'
85972389 | /usr/share/gems/gems/fog-core-2.4.0/lib/fog/core/model.rb:76:in `reload'
85972389 | /usr/share/gems/gems/fog-vsphere-3.7.0/lib/fog/vsphere/models/compute/server.rb:328:in `reload'
85972389 | /usr/share/gems/gems/fog-vsphere-3.7.0/lib/fog/vsphere/models/compute/server.rb:316:in `save'
85972389 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:526:in `create_vm'
85972389 | /usr/share/foreman/app/models/concerns/orchestration/compute.rb:107:in `setCompute'
85972389 | /usr/share/foreman/app/models/concerns/orchestration.rb:227:in `execute'
85972389 | /usr/share/foreman/app/models/concerns/orchestration.rb:152:in `block in process'
85972389 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `each'
85972389 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `process'
85972389 | /usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:137:in `run_callbacks'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:824:in `_run_save_callbacks'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/callbacks.rb:457:in `create_or_update'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/timestamp.rb:126:in `create_or_update'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/persistence.rb:474:in `save'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/validations.rb:47:in `save'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/transactions.rb:298:in `block in save'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/transactions.rb:298:in `save'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/suppressor.rb:44:in `save'
85972389 | /usr/share/foreman/app/models/concerns/foreman/sti.rb:26:in `save'
85972389 | /usr/share/foreman/app/controllers/hosts_controller.rb:100:in `create'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/abstract_controller/base.rb:228:in `process_action'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal/rendering.rb:30:in `process_action'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
85972389 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
85972389 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
85972389 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
85972389 | /usr/share/gems/gems/audited-5.7.0/lib/audited/sweeper.rb:16:in `around'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
85972389 | /usr/share/gems/gems/audited-5.7.0/lib/audited/sweeper.rb:16:in `around'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:137:in `run_callbacks'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/abstract_controller/callbacks.rb:41:in `process_action'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal/rescue.rb:22:in `process_action'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/notifications.rb:203:in `block in instrument'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/notifications.rb:203:in `instrument'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
85972389 | /usr/share/gems/gems/activerecord-6.1.7.9/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/abstract_controller/base.rb:165:in `process'
85972389 | /usr/share/gems/gems/actionview-6.1.7.9/lib/action_view/rendering.rb:39:in `process'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal.rb:190:in `dispatch'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_controller/metal.rb:254:in `dispatch'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/routing/route_set.rb:33:in `serve'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/journey/router.rb:50:in `block in serve'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/journey/router.rb:32:in `each'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/journey/router.rb:32:in `serve'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/routing/route_set.rb:842:in `call'
85972389 | /usr/share/gems/gems/katello-4.14.1/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
85972389 | /usr/share/gems/gems/katello-4.14.1/lib/katello/middleware/event_daemon.rb:10:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/apipie-dsl-2.6.2/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
85972389 | /usr/share/gems/gems/apipie-rails-1.4.2/lib/apipie/static_dispatcher.rb:74:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/static.rb:24:in `call'
85972389 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
85972389 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
85972389 | /usr/share/gems/gems/apipie-rails-1.4.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/tempfile_reaper.rb:15:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/etag.rb:27:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/conditional_get.rb:40:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/head.rb:12:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
85972389 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:266:in `context'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:260:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/cookies.rb:697:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/callbacks.rb:98:in `run_callbacks'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
85972389 | /usr/share/gems/gems/railties-6.1.7.9/lib/rails/rack/logger.rb:37:in `call_app'
85972389 | /usr/share/gems/gems/railties-6.1.7.9/lib/rails/rack/logger.rb:28:in `call'
85972389 | /usr/share/gems/gems/sprockets-rails-3.5.2/lib/sprockets/rails/quiet_assets.rb:17:in `call'
85972389 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/request_id.rb:26:in `call'
85972389 | /usr/share/gems/gems/katello-4.14.1/lib/katello/prevent_json_parsing.rb:12:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/method_override.rb:24:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/runtime.rb:22:in `call'
85972389 | /usr/share/gems/gems/activesupport-6.1.7.9/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/executor.rb:14:in `call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/sendfile.rb:110:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/ssl.rb:77:in `call'
85972389 | /usr/share/gems/gems/actionpack-6.1.7.9/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
85972389 | /usr/share/gems/gems/secure_headers-6.7.0/lib/secure_headers/middleware.rb:11:in `call'
85972389 | /usr/share/gems/gems/railties-6.1.7.9/lib/rails/engine.rb:539:in `call'
85972389 | /usr/share/gems/gems/railties-6.1.7.9/lib/rails/railtie.rb:207:in `public_send'
85972389 | /usr/share/gems/gems/railties-6.1.7.9/lib/rails/railtie.rb:207:in `method_missing'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in `block in call'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `each'
85972389 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `call'
85972389 | /usr/share/gems/gems/puma-6.4.2/lib/puma/configuration.rb:272:in `call'
85972389 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:100:in `block in handle_request'
85972389 | /usr/share/gems/gems/puma-6.4.2/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
85972389 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:99:in `handle_request'
85972389 | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:464:in `process_client'
85972389 | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:245:in `block in run'
85972389 | /usr/share/gems/gems/puma-6.4.2/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
85972389 | /usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-11-28T14:14:18 [W|app|85972389] Rolling back due to a problem: [#<Orchestration::Task:0x000055a5fa956550 @name="Set up compute instance foreman-bug.xxx.xxx", @id="Set up compute instance foreman-bug.xxx.xxx", @status="failed", @priority=3, @action=[#<Host::Managed id: nil, name: "foreman-bug.xxx.xxx", last_compile: nil, last_report: nil, updated_at: nil, created_at: nil, root_pass: nil, architecture_id: 1, operatingsystem_id: 52, ptable_id: 120, medium_id: 24, build: true, comment: "", disk: "", installed_at: nil, model_id: nil, hostgroup_id: 29, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: 1, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: 1, provision_method: "build", grub_pass: nil, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "Grub2 UEFI", initiated_at: nil, build_errors: nil, discovery_rule_id: nil, creator_id: nil>, :setCompute], @created=1732799654.4764826, @timestamp=2024-11-28 13:14:18.820216292 UTC>]
2024-11-28T14:14:18 [I|app|85972389] Processed 1 tasks from queue 'Host::Managed Main', completed 0/8
2024-11-28T14:14:18 [E|app|85972389] Task 'Set up compute instance foreman-bug.xxx.xxx' *failed*
2024-11-28T14:14:18 [E|app|85972389] Task 'Query instance details for foreman-bug.xxx.xxx' *canceled*
2024-11-28T14:14:18 [E|app|85972389] Task 'Deploy TFTP PXELinux config for foreman-bug.xxx.xxx' *canceled*
2024-11-28T14:14:18 [E|app|85972389] Task 'Deploy TFTP PXEGrub2 config for foreman-bug.xxx.xxx' *canceled*
2024-11-28T14:14:18 [E|app|85972389] Task 'Deploy TFTP PXEGrub config for foreman-bug.xxx.xxx' *canceled*
2024-11-28T14:14:18 [E|app|85972389] Task 'Deploy TFTP iPXE config for foreman-bug.xxx.xxx' *canceled*
2024-11-28T14:14:18 [E|app|85972389] Task 'Fetch TFTP boot files for foreman-bug.xxx.xxx' *canceled*
2024-11-28T14:14:18 [E|app|85972389] Task 'Power up compute instance foreman-bug.xxx.xxx' *canceled*
2024-11-28T14:14:18 [D|app|85972389] Unpermitted parameter: :media_selector
2024-11-28T14:14:18 [W|api|85972389] param host[puppetclass_*] has been deprecated in favor of host[puppet_attributes][puppetclass_*]
2024-11-28T14:14:18 [D|app|85972389] Unpermitted parameter: :media_selector
2024-11-28T14:14:18 [W|api|85972389] param host[puppetclass_*] has been deprecated in favor of host[puppet_attributes][puppetclass_*]
2024-11-28T14:14:18 [D|app|85972389] Unpermitted parameter: :media_selector
2024-11-28T14:14:18 [W|api|85972389] param host[puppetclass_*] has been deprecated in favor of host[puppet_attributes][puppetclass_*]
2024-11-28T14:14:18 [E|app|85972389] Failed to save: Failed to create a compute vsphere (VMware) instance foreman-bug.xxx.xxx: InvalidProperty:
85972389 |