Problem:
I already posted this on matrix, but did not recieve any answers there, so I’ll ask for advice here, too.
When we try to change settings on a Foreman host (like changing hostgroup and content view), Foreman sometimes tries to change the VMWare compute instance of that host and adjust it’s CPU and/or RAM. We even had one instance where Foreman tried to create a completely new compute instance on the wrong compute resource, though I am not 100% sure this is related.
After looking around a lot, I guess this could be some form of [Satellite6] Minor change in content host setting triggers update of guest on hypervisor - Red Hat Customer Portal and/or Feature #29813: Don't send memory/cpu attributes to compute_resources if changes are not supported - Foreman, though what really baffles me and at least to me is somewhat contradictive is that inconsistency we are experiencing. Some hosts do not cause this problems, others do. From what I found, the only consistency is that hosts that were provisioned several years ago do not cause this issue, while newer ones do. It also does not matter if the VM Hardware was adjusted after provisioning or not, neither does it make a difference if the host has the hardware as specified in the compute profile or not.
From what we observed, Foreman does not store the HW infos of provisioned VMs anywhere?
Expected outcome:
When no change to the compute attributes is made, Foreman should not try to update the compute instance.
Foreman and Proxy versions:
both 3.2.1 and 3.9.3
Foreman and Proxy plugin versions:
For the 3.9.3 instance:
- foreman-tasks 9.0.4
- foreman_bootdisk 21.2.1
- foreman_expire_hosts 8.2.0
- foreman_hooks 0.3.17
- foreman_puppet 6.2.0
- foreman_remote_execution 12.0.5
- foreman_scc_manager 3.0.0
- foreman_snapshot_management 3.0.0
- foreman_templates 9.4.0
- foreman_webhooks 3.2.2
- katello 4.11.1
- puppetdb_foreman 6.0.2
Distribution and version:
RHEL8.10
Other relevant data:
Logs from one occurance:
2024-07-25T07:40:35 [I|app|cd0ea185] Started PUT "/api/v2/hosts/3368" for 10.164.19.139 at 2024-07-25 07:40:35 +0200
2024-07-25T07:40:35 [I|app|cd0ea185] Processing by Api::V2::HostsController#update as JSON
2024-07-25T07:40:35 [I|app|cd0ea185] Parameters: {"host"=>{"hostgroup_id"=>"203", "config_group_ids"=>"44", "operatingsystem_id"=>"55", "medium_id"=>"62", "ptable_id"=>"95", "build"=>true}, "apiv"=>"v2", "id"=>"3368"}
2024-07-25T07:40:36 [I|app|cd0ea185] Authorized user api-user(api-user)
2024-07-25T07:40:36 [W|app|cd0ea185] DHCP proxy does not report dhcp_filename_ipv4 or dhcp_filename_hostname capability. Foreman will pass boot server 10.165.0.20 as-is, the request might fail.
2024-07-25T07:40:38 [I|app|cd0ea185] Fetching DHCP reservation server.example.com for server.example.com-00:50:56:8a:6e:dd/10.160.253.19
2024-07-25T07:40:41 [I|app|cd0ea185] Fetching DHCP reservation server-nic2.example.com for server-nic2.example.com-00:50:56:8a:89:4f/10.160.160.56
2024-07-25T07:40:43 [I|app|cd0ea185] Loaded compute resource data for networks in 0.437436639 seconds
2024-07-25T07:40:43 [I|app|cd0ea185] Update Compute instance for server.example.com
2024-07-25T07:40:44 [W|app|cd0ea185] Failed to update a compute VCenter K (VMware) instance server.example.com: CpuHotPlugNotSupported: CPU hot plug is not supported for this virtual machine.
2024-07-25T07:40:44 [I|app|cd0ea185] Backtrace for 'Failed to update a compute VCenter K (VMware) instance server.example.com: CpuHotPlugNotSupported: CPU hot plug is not supported for this virtual machine.' error (RbVmomi::Fault): CpuHotPlugNotSupported: CPU hot plug is not supported for this virtual machine.
cd0ea185 | /usr/share/gems/gems/rbvmomi2-3.7.0/lib/rbvmomi/vim/Task.rb:15:in `wait_for_completion'
cd0ea185 | /usr/share/gems/gems/fog-vsphere-3.6.3/lib/fog/vsphere/requests/compute/vm_reconfig_hardware.rb:10:in `vm_reconfig_hardware'
cd0ea185 | /usr/share/gems/gems/fog-vsphere-3.6.3/lib/fog/vsphere/requests/compute/update_vm.rb:35:in `update_vm'
cd0ea185 | /usr/share/gems/gems/fog-vsphere-3.6.3/lib/fog/vsphere/models/compute/server.rb:304:in `save'
cd0ea185 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:530:in `save_vm'
cd0ea185 | /usr/share/foreman/app/models/concerns/orchestration/compute.rb:243:in `setComputeUpdate'
cd0ea185 | /usr/share/foreman/app/models/concerns/orchestration.rb:227:in `execute'
cd0ea185 | /usr/share/foreman/app/models/concerns/orchestration.rb:152:in `block in process'
cd0ea185 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `each'
cd0ea185 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `process'
cd0ea185 | /usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:137:in `run_callbacks'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:824:in `_run_save_callbacks'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/callbacks.rb:457:in `create_or_update'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/timestamp.rb:126:in `create_or_update'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/persistence.rb:474:in `save'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/validations.rb:47:in `save'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/transactions.rb:298:in `block in save'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/transactions.rb:298:in `save'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/suppressor.rb:44:in `save'
cd0ea185 | /usr/share/foreman/app/models/concerns/foreman/sti.rb:26:in `save'
cd0ea185 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:164:in `update'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/base.rb:228:in `process_action'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
cd0ea185 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
cd0ea185 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
cd0ea185 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
cd0ea185 | /usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
cd0ea185 | /usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:137:in `run_callbacks'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:203:in `block in instrument'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:203:in `instrument'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
cd0ea185 | /usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/base.rb:165:in `process'
cd0ea185 | /usr/share/gems/gems/actionview-6.1.7.7/lib/action_view/rendering.rb:39:in `process'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal.rb:190:in `dispatch'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal.rb:254:in `dispatch'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/mapper.rb:49:in `serve'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/journey/router.rb:50:in `block in serve'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/journey/router.rb:32:in `each'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/journey/router.rb:32:in `serve'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:842:in `call'
cd0ea185 | /usr/share/gems/gems/katello-4.11.1/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
cd0ea185 | /usr/share/gems/gems/katello-4.11.1/lib/katello/middleware/event_daemon.rb:10:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/apipie-dsl-2.6.1/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
cd0ea185 | /usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/static_dispatcher.rb:68:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in `call'
cd0ea185 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
cd0ea185 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
cd0ea185 | /usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/conditional_get.rb:40:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
cd0ea185 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/cookies.rb:697:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:98:in `run_callbacks'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
cd0ea185 | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:37:in `call_app'
cd0ea185 | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:28:in `call'
cd0ea185 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
cd0ea185 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
cd0ea185 | /usr/share/gems/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/request_id.rb:26:in `call'
cd0ea185 | /usr/share/gems/gems/katello-4.11.1/lib/katello/prevent_json_parsing.rb:12:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
cd0ea185 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/ssl.rb:77:in `call'
cd0ea185 | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
cd0ea185 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
cd0ea185 | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/engine.rb:539:in `call'
cd0ea185 | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
cd0ea185 | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
cd0ea185 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
cd0ea185 | /usr/share/gems/gems/puma-6.4.0/lib/puma/configuration.rb:272:in `call'
cd0ea185 | /usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:100:in `block in handle_request'
cd0ea185 | /usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
cd0ea185 | /usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:99:in `handle_request'
cd0ea185 | /usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:443:in `process_client'
cd0ea185 | /usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:241:in `block in run'
cd0ea185 | /usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
cd0ea185 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-07-25T07:40:44 [W|app|cd0ea185] Rolling back due to a problem: [#<Orchestration::Task:0x00007f320dc5bea8 @name="Compute resource update for server.example.com", @id="Compute resource update for server.example.com", @status="failed", @priority=7, @action=[#<Host::Managed id: 3368, name: "server.example.com", last_compile: "2024-07-24 23:04:59.000000000 +0200", last_report: nil, updated_at: "2024-07-24 23:04:59.489287000 +0200", created_at: "2024-07-24 13:59:20.202064000 +0200", root_pass: nil, architecture_id: 1, operatingsystem_id: 55, ptable_id: 95, medium_id: 62, build: true, comment: "", disk: "", installed_at: "2024-07-24 14:09:20.725742000 +0200", model_id: 1, hostgroup_id: 203, owner_id: 19, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: "500a4c34-48a6-3f59-da56-9ee8d2512939", compute_resource_id: 4, puppet_proxy_id: 5, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: 1, provision_method: "build", grub_pass: nil, global_status: 1, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", initiated_at: nil, build_errors: nil, expired_on: nil, creator_id: 19>, :setComputeUpdate], @created=1721886043.2512074, @timestamp=2024-07-25 05:40:44.734439136 UTC>]
2024-07-25T07:40:44 [I|app|cd0ea185] Processed 1 tasks from queue 'Host::Managed Main', completed 0/7
2024-07-25T07:40:44 [E|app|cd0ea185] Task 'Compute resource update for server.example.com' *failed*
2024-07-25T07:40:44 [E|app|cd0ea185] Task 'Deploy TFTP PXELinux config for server.example.com' *canceled*
2024-07-25T07:40:44 [E|app|cd0ea185] Task 'Deploy TFTP PXEGrub2 config for server.example.com' *canceled*
2024-07-25T07:40:44 [E|app|cd0ea185] Task 'Deploy TFTP PXEGrub config for server.example.com' *canceled*
2024-07-25T07:40:44 [E|app|cd0ea185] Task 'Deploy TFTP iPXE config for server.example.com' *canceled*
2024-07-25T07:40:44 [E|app|cd0ea185] Task 'Fetch TFTP boot files for server.example.com' *canceled*
2024-07-25T07:40:44 [E|app|cd0ea185] Task 'Mark Content Host Statuses as Unknown for server.example.com' *canceled*
2024-07-25T07:40:44 [E|app|cd0ea185] Unprocessable entity Host::Managed (id: 3368):