Problem:
We have since a long time now (many releases back) experienced performance issues when creating new host in certain hostgroups. We thought this could be related to how many hosts lived inside a hostgroup, with our biggest one housing 1200+ hosts. After some digging around, we found out that the same behaviour occured with a hostgroup of 200+ hosts. Creating a new host can take up to 15 minutes (always similar timeframe using the same hostgroup) while in comparison with a different hostgroup of 60 hosts it takes about 2 minutes. Not at all certain that this has anything to do with the size of the hostgroup or if it’s a coincidence. Do you know what could be the cause of this or where to further troubleshoot?
Expected outcome:
We need to be able to create a lot more hosts on a daily basis compared to now.
Foreman and Proxy versions:
Foreman 3.4 and Katello 4.6
Distribution and version:
RHEL 8
Other relevant data:
Create host inside hostgroup with 1200+ hosts. (production.log)
2022-10-13T10:41:12 [I|app|15d1102e] Parameters: {"utf8"=>"✓", "authenticity_token"=>"xxxxxxxxxxxxxxxxxxx", "host"=>{"name"=>"slowhost01", "organization_id"=>"1", "location_id"=>"2", "hostgroup_id"=>"370", "compute_resource_id"=>"1", "content_facet_attributes"=>{"lifecycle_environment_id"=>"1", "content_view_id"=>"1", "content_source_id"=>"1", "kickstart_repository_id"=>"428"}, "puppet_attributes"=>{"environment_id"=>"5324"}, "config_group_ids"=>[""], "puppetclass_ids"=>[""], "managed"=>"true", "progress_report_id"=>"[FILTERED]", "type"=>"Host::Managed", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", "type"=>"Nic::Managed", "identifier"=>"", "name"=>"slowhost01", "domain_id"=>"2", "subnet_id"=>"17", "ip6"=>"", "managed"=>"1", "primary"=>"1", "provision"=>"1", "execution"=>"1", "virtual"=>"0", "tag"=>"", "attached_to"=>"", "compute_attributes"=>{"type"=>"VirtualVmxnet3", "network"=>"dvportgroup-15350"}}}, "compute_attributes"=>{"cpus"=>"8", "corespersocket"=>"1", "memory_mb"=>"49152", "firmware"=>"efi", "cluster"=>"Prod", "resource_pool"=>"Resources", "path"=>"/Datacenters/Lunda/vm", "guest_id"=>"rhel8_64Guest", "hardware_version"=>"Default", "memoryHotAddEnabled"=>"1", "cpuHotAddEnabled"=>"1", "add_cdrom"=>"0", "boot_order"=>["network", "disk"], "start"=>"1", "annotation"=>"", "scsi_controllers"=>"{\"scsiControllers\":[{\"type\":\"ParaVirtualSCSIController\",\"key\":1000}],\"volumes\":[{\"thin\":true,\"name\":\"Hard disk\",\"mode\":\"persistent\",\"controllerKey\":1000,\"storagePod\":\"cmpl03\",\"eagerZero\":false,\"sizeGb\":600}]}", "image_id"=>"50054bf6-c731-801e-b4c9-6de784fca259"}, "architecture_id"=>"1", "operatingsystem_id"=>"56", "provision_method"=>"image", "build"=>"1", "medium_id"=>"", "ptable_id"=>"208", "pxe_loader"=>"Grub2 UEFI", "disk"=>"", "root_pass"=>"[FILTERED]", "is_owned_by"=>"56-Users", "enabled"=>"1", "model_id"=>"", "comment"=>"", "overwrite"=>"false"}, "media_selector"=>"synced_content"}
2022-10-13T10:55:38 [I|app|15d1102e] Rendering UserData template for slowhost01.domain.com
2022-10-13T10:55:39 [I|app|15d1102e] Adding Compute instance for slowhost01.domain.com
2022-10-13T10:55:55 [I|app|15d1102e] Create DHCP reservation slowhost01.domain.com for slowhost01.domain.com-00:50:56:85:8b:9f/10.83.68.181
2022-10-13T10:55:55 [I|app|15d1102e] Add DNS A record for slowhost01.domain.com/10.83.68.181
2022-10-13T10:56:21 [I|app|15d1102e] Add DNS PTR record for 10.83.68.181/slowhost01.domain.com
2022-10-13T10:56:50 [I|app|15d1102e] Powering up Compute instance for slowhost01.domain.com
2022-10-13T10:56:53 [I|aud|15d1102e] Nic::Managed (20987) create event on name slowhost01.domain.com
2022-10-13T10:56:53 [I|aud|15d1102e] Host::Base (18758) create event on name slowhost01.domain.com
2022-10-13T10:56:54 [I|app|15d1102e] Remove puppet certificate for slowhost01.domain.com
2022-10-13T10:56:54 [I|app|15d1102e] Adding autosign entry for slowhost01.domain.com
2022-10-13T10:56:54 [I|app|15d1102e] Redirected to https://katello.domain.com/new/hosts/slowhost01.domain.com
2022-10-13T10:57:26 [I|app|10170b25] Started GET "/node/slowhost01.domain.com?format=yml" for 10.87.1.14 at 2022-10-13 10:57:26 +0200
2022-10-13T10:57:26 [I|app|10170b25] Parameters: {"name"=>"slowhost01.domain.com"}
2022-10-13T10:57:44 [I|app|36bed7ab] Parameters: {"facts"=>"[FILTERED]", "name"=>"slowhost01.domain.com", "certname"=>"slowhost01.domain.com", "apiv"=>"v2", "host"=>{"certname"=>"slowhost01.domain.com", "name"=>"slowhost01.domain.com"}}
Create host inside hostgroup with 60+ hosts. (production.log)
2022-10-13T10:37:23 [I|app|cba2d852] Parameters: {"utf8"=>"✓", "authenticity_token"=>"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "host"=>{"name"=>"fasthost01", "organization_id"=>"1", "location_id"=>"2", "hostgroup_id"=>"4", "compute_resource_id"=>"1", "content_facet_attributes"=>{"lifecycle_environment_id"=>"1", "content_view_id"=>"1", "content_source_id"=>"1", "kickstart_repository_id"=>"237"}, "config_group_ids"=>[""], "puppetclass_ids"=>[""], "managed"=>"true", "progress_report_id"=>"[FILTERED]", "type"=>"Host::Managed", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", "type"=>"Nic::Managed", "identifier"=>"", "name"=>"fasthost01", "domain_id"=>"1", "subnet_id"=>"1", "ip"=>"10.87.1.66", "ip6"=>"", "managed"=>"1", "primary"=>"1", "provision"=>"1", "execution"=>"1", "virtual"=>"0", "tag"=>"", "attached_to"=>"", "compute_attributes"=>{"type"=>"VirtualVmxnet3", "network"=>"dvportgroup-4930"}}}, "compute_attributes"=>{"cpus"=>"2", "corespersocket"=>"1", "memory_mb"=>"4096", "firmware"=>"efi", "cluster"=>"Prod", "resource_pool"=>"Resources", "path"=>"/Datacenters/Lunda/vm", "guest_id"=>"rhel8_64Guest", "hardware_version"=>"Default", "memoryHotAddEnabled"=>"1", "cpuHotAddEnabled"=>"1", "add_cdrom"=>"0", "boot_order"=>["network", "disk"], "start"=>"1", "annotation"=>"", "scsi_controllers"=>"{\"scsiControllers\":[{\"type\":\"ParaVirtualSCSIController\",\"key\":1000}],\"volumes\":[{\"thin\":true,\"name\":\"Hard disk\",\"mode\":\"persistent\",\"controllerKey\":1000,\"sizeGb\":600,\"eagerZero\":\"false\",\"storagePod\":\"cmpl03\"}]}", "image_id"=>"50054bf6-c731-801e-b4c9-6de784fca259"}, "architecture_id"=>"1", "operatingsystem_id"=>"56", "provision_method"=>"image", "build"=>"1", "medium_id"=>"", "ptable_id"=>"208", "pxe_loader"=>"Grub2 UEFI", "disk"=>"", "root_pass"=>"[FILTERED]", "is_owned_by"=>"56-Users", "enabled"=>"1", "model_id"=>"", "comment"=>"", "overwrite"=>"false"}, "media_selector"=>"synced_content"}
2022-10-13T10:37:23 [I|app|cba2d852] Rendering UserData template for fasthost01.domain.com
2022-10-13T10:37:24 [I|app|cba2d852] Adding Compute instance for fasthost01.domain.com
2022-10-13T10:37:38 [I|app|cba2d852] Create DHCP reservation fasthost01.domain.com for fasthost01.domain.com-00:50:56:85:d7:57/10.87.1.66
2022-10-13T10:37:38 [I|app|cba2d852] Add DNS A record for fasthost01.domain.com/10.87.1.66
2022-10-13T10:38:04 [I|app|cba2d852] Add DNS PTR record for 10.87.1.66/fasthost01.domain.com
2022-10-13T10:38:35 [I|app|cba2d852] Powering up Compute instance for fasthost01.domain.com
2022-10-13T10:38:38 [I|aud|cba2d852] Nic::Managed (20986) create event on name fasthost01.domain.com
2022-10-13T10:38:38 [I|aud|cba2d852] Host::Base (18757) create event on name fasthost01.domain.com
2022-10-13T10:38:38 [I|app|cba2d852] Remove puppet certificate for fasthost01.domain.com
2022-10-13T10:38:38 [I|app|cba2d852] Adding autosign entry for fasthost01.domain.com
2022-10-13T10:38:38 [I|app|cba2d852] Redirected to https://katello.domain.com/new/hosts/fasthost01.domain.com
2022-10-13T10:38:38 [I|app|654b9d79] Started GET "/new/hosts/fasthost01.domain.com" for 172.16.38.210 at 2022-10-13 10:38:38 +0200
2022-10-13T10:38:38 [I|app|654b9d79] Parameters: {"id"=>"fasthost01.domain.com"}
2022-10-13T10:38:38 [I|app|79cabc95] Started GET "/api/hosts/fasthost01.domain.com" for 172.16.38.210 at 2022-10-13 10:38:38 +0200
2022-10-13T10:38:38 [I|app|79cabc95] Parameters: {"apiv"=>"v2", "id"=>"fasthost01.domain.com"}
2022-10-13T10:38:38 [I|app|6b953b73] Started GET "/hosts/fasthost01.domain.com/statuses" for 172.16.38.210 at 2022-10-13 10:38:38 +0200
2022-10-13T10:38:38 [I|app|452f36a0] Started GET "/api/hosts/fasthost01.domain.com/power?timeout=30" for 172.16.38.210 at 2022-10-13 10:38:38 +0200
2022-10-13T10:38:38 [I|app|97b37e03] Started GET "/api/audits?search=host%3Dfasthost01.domain.com&per_page=3" for 172.16.38.210 at 2022-10-13 10:38:38 +0200
2022-10-13T10:38:38 [I|app|6b953b73] Parameters: {"id"=>"fasthost01.domain.com"}
2022-10-13T10:38:38 [I|app|97b37e03] Parameters: {"search"=>"host=fasthost01.domain.com", "per_page"=>"3", "apiv"=>"v2"}
2022-10-13T10:38:38 [I|app|452f36a0] Parameters: {"timeout"=>"30", "apiv"=>"v2", "id"=>"fasthost01.domain.com"}