Foreman slow performance when clicking hosts

Support
1

Problem:
My Foreman Server is managing only 27 servers. I am using Katello and the Ansible-Plugin.
When I click between “All Hosts” and “Content Hosts”, I consistently have to wait for about 20-30s for each of the Views.
During wait time, I can see two processes “rails” spiking up to 100%. There CPU load on the Browser side is idle.
The server has 4x CPU, 20GB RAM, plentyful disk.
I noticed a massive slowdown, after I started to frequently collect facts using Ansible (every 2 mins).
Clicking on other items in the GUI is fine (e.g. “Monitor”/“Jobs” or “Content”/“Products” etc).

Expected outcome:
Wait time below 2s

Foreman and Proxy versions:
Foreman: 3.7.0

Foreman and Proxy plugin versions:
foreman-tasks: 8.1.1
foreman_ansible: 12.0.6
foreman_remote_execution: 10.0.7
katello: 4.9.1

Distribution and version:
Rocky Linux 8.8

Other relevant data:

# cat /var/log/foreman/production.log | grep ' Completed ' | perl -e '{while(<>){if(/ in (\d+)ms /){ if($1>3000){print} }}}' | tail -20
2023-09-22T17:25:53 [I|app|4e2baf2d] Completed 200 OK in 32105ms (Views: 25334.6ms | ActiveRecord: 6693.3ms | Allocations: 16981358)
2023-09-22T17:26:01 [I|app|bbe62476] Completed 200 OK in 4122ms (Views: 0.2ms | ActiveRecord: 2265.0ms | Allocations: 5118065)
2023-09-22T17:26:01 [I|app|19e358c1] Completed 302 Found in 3941ms (ActiveRecord: 254.9ms | Allocations: 5145704)
2023-09-22T17:28:38 [I|app|1f00eeba] Completed 200 OK in 17539ms (Views: 16021.5ms | ActiveRecord: 1470.2ms | Allocations: 11986826)
2023-09-22T17:29:08 [I|app|87277620] Completed 200 OK in 23515ms (Views: 21965.8ms | ActiveRecord: 1513.6ms | Allocations: 16963953)
2023-09-22T17:29:10 [I|app|32dbba78] Completed 200 OK in 25093ms (Views: 23483.5ms | ActiveRecord: 1578.2ms | Allocations: 16970110)
2023-09-22T17:32:53 [I|app|4ec3abe8] Completed 200 OK in 17211ms (Views: 15650.4ms | ActiveRecord: 1438.6ms | Allocations: 11991516)
2023-09-22T17:40:51 [I|app|3e539aa1] Completed 200 OK in 23667ms (Views: 21955.1ms | ActiveRecord: 1679.8ms | Allocations: 16921396)
2023-09-22T17:40:52 [I|app|7ce5c42e] Completed 200 OK in 24785ms (Views: 23142.8ms | ActiveRecord: 1578.6ms | Allocations: 16923736)
2023-09-22T17:40:54 [I|app|28b67067] Completed 200 OK in 23949ms (Views: 22087.6ms | ActiveRecord: 1823.1ms | Allocations: 17160661)
2023-09-22T17:43:15 [I|app|430fd66b] Completed 200 OK in 17564ms (Views: 16131.0ms | ActiveRecord: 1386.5ms | Allocations: 11936421)
2023-09-22T17:43:43 [I|app|d8927418] Completed 200 OK in 23602ms (Views: 22046.0ms | ActiveRecord: 1522.5ms | Allocations: 16943744)
2023-09-22T17:43:44 [I|app|b5c39464] Completed 200 OK in 24537ms (Views: 23090.5ms | ActiveRecord: 1419.1ms | Allocations: 16930745)
2023-09-22T17:44:35 [I|app|14778cb8] Completed 200 OK in 16811ms (Views: 15369.1ms | ActiveRecord: 1384.0ms | Allocations: 11946665)
2023-09-22T17:45:48 [I|app|b3a43d6c] Completed 200 OK in 7337ms (Views: 6314.3ms | ActiveRecord: 1016.7ms | Allocations: 5707994)
2023-09-22T17:46:39 [I|app|e5735c16] Completed 200 OK in 25299ms (Views: 23664.9ms | ActiveRecord: 1593.2ms | Allocations: 16944205)
2023-09-22T18:07:38 [I|app|b9b4ef5a] Completed 200 OK in 23086ms (Views: 21518.9ms | ActiveRecord: 1540.0ms | Allocations: 16965561)
2023-09-22T18:07:38 [I|app|7e909b77] Completed 200 OK in 23601ms (Views: 22079.1ms | ActiveRecord: 1479.2ms | Allocations: 16945945)
2023-09-22T18:10:04 [I|app|5473283d] Completed 200 OK in 23295ms (Views: 21804.8ms | ActiveRecord: 1466.6ms | Allocations: 16890232)
2023-09-22T18:10:55 [I|app|58050a93] Completed 200 OK in 7599ms (Views: 7579.7ms | ActiveRecord: 4.7ms | Allocations: 47507)

# cat production.log_excerpt  
2023-09-22T18:17:39 [I|app|ae089d6d] Started HEAD "/" for 10.11.12.41 at 2023-09-22 18:17:39 +0200
2023-09-22T18:17:39 [I|app|ae089d6d] Processing by DashboardController#index as HTML
2023-09-22T18:17:39 [I|app|ae089d6d] Redirected to https://foreman.my.org/users/login
2023-09-22T18:17:39 [I|app|ae089d6d] Filter chain halted as :require_login rendered or redirected
2023-09-22T18:17:39 [I|app|ae089d6d] Completed 302 Found in 17ms (ActiveRecord: 13.2ms | Allocations: 754)
2023-09-22T18:17:42 [I|app|bb0e64ed] Started HEAD "/" for 10.11.12.42 at 2023-09-22 18:17:42 +0200
2023-09-22T18:17:42 [I|app|bb0e64ed] Processing by DashboardController#index as HTML
2023-09-22T18:17:42 [I|app|bb0e64ed] Redirected to https://foreman.my.org/users/login
2023-09-22T18:17:42 [I|app|bb0e64ed] Filter chain halted as :require_login rendered or redirected
2023-09-22T18:17:42 [I|app|bb0e64ed] Completed 302 Found in 3ms (ActiveRecord: 1.1ms | Allocations: 755)
2023-09-22T18:17:42 [I|app|b8c0805c] Started HEAD "/" for 10.11.12.41 at 2023-09-22 18:17:42 +0200
2023-09-22T18:17:42 [I|app|b8c0805c] Processing by DashboardController#index as HTML
2023-09-22T18:17:42 [I|app|b8c0805c] Redirected to https://foreman.my.org/users/login
2023-09-22T18:17:42 [I|app|b8c0805c] Filter chain halted as :require_login rendered or redirected
2023-09-22T18:17:42 [I|app|b8c0805c] Completed 302 Found in 3ms (ActiveRecord: 1.1ms | Allocations: 754)
2023-09-22T18:17:45 [I|app|58e3c506] Started HEAD "/" for 10.11.12.42 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|58e3c506] Processing by DashboardController#index as HTML
2023-09-22T18:17:45 [I|app|58e3c506] Redirected to https://foreman.my.org/users/login
2023-09-22T18:17:45 [I|app|58e3c506] Filter chain halted as :require_login rendered or redirected
2023-09-22T18:17:45 [I|app|58e3c506] Completed 302 Found in 4ms (ActiveRecord: 1.3ms | Allocations: 754)
2023-09-22T18:17:45 [I|app|e7a07304]   Rendered hosts/index.html.erb within layouts/application (Duration: 23325.8ms | Allocations: 16904150)
2023-09-22T18:17:45 [I|app|e7a07304]   Rendered layouts/base.html.erb (Duration: 16.7ms | Allocations: 20284)
2023-09-22T18:17:45 [I|app|e7a07304]   Rendered layout layouts/application.html.erb (Duration: 23346.0ms | Allocations: 16928041)
2023-09-22T18:17:45 [I|app|e7a07304] Completed 200 OK in 23425ms (Views: 21831.5ms | ActiveRecord: 1556.7ms | Allocations: 16952622)
2023-09-22T18:17:45 [I|app|df127894] Started HEAD "/" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|df127894] Processing by DashboardController#index as HTML
2023-09-22T18:17:45 [I|app|df127894] Redirected to https://foreman.my.org/users/login
2023-09-22T18:17:45 [I|app|df127894] Filter chain halted as :require_login rendered or redirected
2023-09-22T18:17:45 [I|app|df127894] Completed 302 Found in 4ms (ActiveRecord: 2.8ms | Allocations: 757)
2023-09-22T18:17:45 [I|app|2ddc9595] Started GET "/notification_recipients" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|2ddc9595] Processing by NotificationRecipientsController#index as JSON
2023-09-22T18:17:45 [I|app|2ddc9595] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.8ms | Allocations: 1599)
2023-09-22T18:17:45 [I|app|16b830f8] Started GET "/hosts/auto_complete_search?search=" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|16b830f8] Processing by HostsController#auto_complete_search as JSON
2023-09-22T18:17:45 [I|app|16b830f8]   Parameters: {"search"=>""}
2023-09-22T18:17:45 [I|app|16b830f8] Completed 200 OK in 9ms (Views: 2.0ms | ActiveRecord: 1.4ms | Allocations: 11018)
2023-09-22T18:17:45 [I|app|f8038052] Started GET "/api/v2/hosts/cvkasv2545sfgdf01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|f8038052] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:45 [I|app|f8038052]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv2545sfgdf01.my.org"}
2023-09-22T18:17:45 [I|app|f8038052] Completed 200 OK in 10ms (Views: 0.1ms | ActiveRecord: 2.3ms | Allocations: 3339)
2023-09-22T18:17:45 [I|app|4a6181c0] Started GET "/api/v2/hosts/cvkasv2545sfgdf02.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|4a6181c0] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:45 [I|app|4a6181c0]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv2545sfgdf02.my.org"}
2023-09-22T18:17:45 [I|app|4a6181c0] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.9ms | Allocations: 3339)
2023-09-22T18:17:45 [I|app|e41c1ffc] Started GET "/api/v2/hosts/cvkasv4682sfgdf01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|e41c1ffc] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:45 [I|app|e41c1ffc]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sfgdf01.my.org"}
2023-09-22T18:17:45 [I|app|e41c1ffc] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 3.2ms | Allocations: 3338)
2023-09-22T18:17:45 [I|app|2f755644] Started GET "/api/v2/hosts/cvkasv4682sfgdf02.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|2f755644] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:45 [I|app|2f755644]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sfgdf02.my.org"}
2023-09-22T18:17:45 [I|app|2f755644] Completed 200 OK in 10ms (Views: 0.2ms | ActiveRecord: 2.6ms | Allocations: 3387)
2023-09-22T18:17:45 [I|app|ad49ec1b] Started GET "/api/v2/hosts/cvkasv4682sbgsl1.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|ad49ec1b] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:45 [I|app|ad49ec1b]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sbgsl1.my.org"}
2023-09-22T18:17:45 [I|app|ad49ec1b] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 2.0ms | Allocations: 3338)
2023-09-22T18:17:45 [I|app|d16a5759]   Rendered hosts/index.html.erb within layouts/application (Duration: 24233.1ms | Allocations: 16925320)
2023-09-22T18:17:45 [I|app|beffe840] Started GET "/api/v2/hosts/cvkasv4682sbgsl2.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:45 +0200
2023-09-22T18:17:45 [I|app|beffe840] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:45 [I|app|beffe840]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sbgsl2.my.org"}
2023-09-22T18:17:45 [I|app|beffe840] Completed 200 OK in 10ms (Views: 0.1ms | ActiveRecord: 2.6ms | Allocations: 3341)
2023-09-22T18:17:45 [I|app|d16a5759]   Rendered layouts/base.html.erb (Duration: 15.1ms | Allocations: 21134)
2023-09-22T18:17:45 [I|app|d16a5759]   Rendered layout layouts/application.html.erb (Duration: 24255.3ms | Allocations: 16959669)
2023-09-22T18:17:45 [I|app|d16a5759] Completed 200 OK in 24318ms (Views: 22834.3ms | ActiveRecord: 1451.0ms | Allocations: 16977280)
2023-09-22T18:17:46 [I|app|504c937d] Started GET "/api/v2/hosts/cvkasv4682sbgsl3.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|504c937d] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|504c937d]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sbgsl3.my.org"}
2023-09-22T18:17:46 [I|app|504c937d] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 2.0ms | Allocations: 3354)
2023-09-22T18:17:46 [I|app|b669281f] Started GET "/api/v2/hosts/cvkasv4682sbgsd01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|b669281f] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|b669281f]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sbgsd01.my.org"}
2023-09-22T18:17:46 [I|app|b669281f] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.7ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|af6b66c9] Started GET "/api/v2/hosts/cvkasv4682sbgsd02.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|af6b66c9] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|af6b66c9]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sbgsd02.my.org"}
2023-09-22T18:17:46 [I|app|af6b66c9] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 1.7ms | Allocations: 3339)
2023-09-22T18:17:46 [I|app|1a3a30b7] Started GET "/api/v2/hosts/cvkasv4682sbgsd03.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|1a3a30b7] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|1a3a30b7]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682sbgsd03.my.org"}
2023-09-22T18:17:46 [I|app|1a3a30b7] Completed 200 OK in 10ms (Views: 0.1ms | ActiveRecord: 2.7ms | Allocations: 3341)
2023-09-22T18:17:46 [I|app|5ecabb3e] Started GET "/api/v2/hosts/cvkasv4682slfgf01b.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|5ecabb3e] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|5ecabb3e]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682slfgf01b.my.org"}
2023-09-22T18:17:46 [I|app|5ecabb3e] Completed 200 OK in 7ms (Views: 0.1ms | ActiveRecord: 0.9ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|5abb3519] Started GET "/api/v2/hosts/cvkasv4682slfgf01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|5abb3519] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|5abb3519]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682slfgf01.my.org"}
2023-09-22T18:17:46 [I|app|5abb3519] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.9ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|91381671] Started GET "/api/v2/hosts/cvkasv4682nfs01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|91381671] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|91381671]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682nfs01.my.org"}
2023-09-22T18:17:46 [I|app|91381671] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 1.6ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|6931b372] Started GET "/api/v2/hosts/cvkasv4682ns1.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|6931b372] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|6931b372]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682ns1.my.org"}
2023-09-22T18:17:46 [I|app|6931b372] Completed 200 OK in 7ms (Views: 0.1ms | ActiveRecord: 1.7ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|317096d3] Started GET "/api/v2/hosts/cvkasv4682ns2.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|317096d3] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|317096d3]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv4682ns2.my.org"}
2023-09-22T18:17:46 [I|app|317096d3] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.8ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|20f41a37] Started GET "/api/v2/hosts/cvkasv9843sfghd01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|20f41a37] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|20f41a37]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv9843sfghd01.my.org"}
2023-09-22T18:17:46 [I|app|20f41a37] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 2.0ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|0f94c4b1] Started GET "/api/v2/hosts/foreman.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|0f94c4b1] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|0f94c4b1]   Parameters: {"apiv"=>"v2", "id"=>"foreman.my.org"}
2023-09-22T18:17:46 [I|app|9e7df390] Started GET "/api/v2/hosts/cvkasv9843stfgg01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|0f94c4b1] Completed 200 OK in 10ms (Views: 0.1ms | ActiveRecord: 2.1ms | Allocations: 3340)
2023-09-22T18:17:46 [I|app|9e7df390] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|9e7df390]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv9843stfgg01.my.org"}
2023-09-22T18:17:46 [I|app|9e7df390] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.9ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|49d4dcbb] Started GET "/api/v2/hosts/cvkasv9843stfgg02.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|49d4dcbb] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|49d4dcbb]   Parameters: {"apiv"=>"v2", "id"=>"cvkasv9843stfgg02.my.org"}
2023-09-22T18:17:46 [I|app|49d4dcbb] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 1.4ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|1609b8bd] Started GET "/api/v2/hosts/cfgdgv6342sdfsg01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|1609b8bd] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|1609b8bd]   Parameters: {"apiv"=>"v2", "id"=>"cfgdgv6342sdfsg01.my.org"}
2023-09-22T18:17:46 [I|app|7172c34d] Started GET "/api/v2/hosts/cfgdgv6342sdfsg02.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|1609b8bd] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 1.2ms | Allocations: 3343)
2023-09-22T18:17:46 [I|app|7172c34d] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|7172c34d]   Parameters: {"apiv"=>"v2", "id"=>"cfgdgv6342sdfsg02.my.org"}
2023-09-22T18:17:46 [I|app|7172c34d] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 2.2ms | Allocations: 3345)
2023-09-22T18:17:46 [I|app|b78b67a8] Started GET "/api/v2/hosts/cghggv3213sfgdf01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|b78b67a8] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|b78b67a8]   Parameters: {"apiv"=>"v2", "id"=>"cghggv3213sfgdf01.my.org"}
2023-09-22T18:17:46 [I|app|b78b67a8] Completed 200 OK in 12ms (Views: 0.1ms | ActiveRecord: 2.4ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|59035c98] Started GET "/api/v2/hosts/cghggv3213sfgdf02.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|59035c98] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|59035c98]   Parameters: {"apiv"=>"v2", "id"=>"cghggv3213sfgdf02.my.org"}
2023-09-22T18:17:46 [I|app|59035c98] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 1.4ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|b770abe5] Started GET "/api/v2/hosts/cghggv3213slfgf01b.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|b770abe5] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|b770abe5]   Parameters: {"apiv"=>"v2", "id"=>"cghggv3213slfgf01b.my.org"}
2023-09-22T18:17:46 [I|app|b770abe5] Completed 200 OK in 12ms (Views: 0.1ms | ActiveRecord: 2.3ms | Allocations: 3339)
2023-09-22T18:17:46 [I|app|0fd589bb] Started GET "/api/v2/hosts/cghggv3213slfgf01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|0fd589bb] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|0fd589bb]   Parameters: {"apiv"=>"v2", "id"=>"cghggv3213slfgf01.my.org"}
2023-09-22T18:17:46 [I|app|0fd589bb] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.9ms | Allocations: 3338)
2023-09-22T18:17:46 [I|app|9a2bce54] Started GET "/api/v2/hosts/cghggv3213sfgfh01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|9a2bce54] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|9a2bce54]   Parameters: {"apiv"=>"v2", "id"=>"cghggv3213sfgfh01.my.org"}
2023-09-22T18:17:46 [I|app|9a2bce54] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 2.3ms | Allocations: 3339)
2023-09-22T18:17:46 [I|app|41cd2dbd] Started GET "/api/v2/hosts/cdfvgv7565swefg01.my.org/power" for 10.11.12.41 at 2023-09-22 18:17:46 +0200
2023-09-22T18:17:46 [I|app|41cd2dbd] Processing by Api::V2::HostsController#power_status as JSON
2023-09-22T18:17:46 [I|app|41cd2dbd]   Parameters: {"apiv"=>"v2", "id"=>"cdfvgv7565swefg01.my.org"}
2023-09-22T18:17:46 [I|app|41cd2dbd] Completed 200 OK in 12ms (Views: 0.1ms | ActiveRecord: 2.3ms | Allocations: 3338)
2023-09-22T18:17:48 [I|app|0fb843ef] Started HEAD "/" for 10.11.12.42 at 2023-09-22 18:17:48 +0200
2023-09-22T18:17:48 [I|app|0fb843ef] Processing by DashboardController#index as HTML
2023-09-22T18:17:48 [I|app|0fb843ef] Redirected to https://foreman.my.org/users/login
2023-09-22T18:17:48 [I|app|0fb843ef] Filter chain halted as :require_login rendered or redirected
2023-09-22T18:17:48 [I|app|0fb843ef] Completed 302 Found in 5ms (ActiveRecord: 1.8ms | Allocations: 756)
2023-09-22T18:17:48 [I|app|284330f3] Started HEAD "/" for 10.11.12.41 at 2023-09-22 18:17:48 +0200
2023-09-22T18:17:48 [I|app|284330f3] Processing by DashboardController#index as HTML
2023-09-22T18:17:48 [I|app|284330f3] Redirected to https://foreman.my.org/users/login
2023-09-22T18:17:48 [I|app|284330f3] Filter chain halted as :require_login rendered or redirected
2023-09-22T18:17:48 [I|app|284330f3] Completed 302 Found in 5ms (ActiveRecord: 1.7ms | Allocations: 754)
2

Hi! Maybe you have way too much reports or facts. Try to reduce these by running foreman-rake commands.
You can list all foreman-rake commands by running:

foreman-rake --tasks

3

Thanks for suggesting.

I could only find rake facts:clean.

After running the following command, it is still as slow as before.

# foreman-rake facts:clean
Starting orphaned facts clean up
Finished, cleaned 37648 facts

I wasn’t able to fix the issue. :slightly_frowning_face:

Also, I’d like to note that I didn’t deliberately add any facts.
I was just running the Ansible Roles.

The majority of my hosts has ~115 Facts Values.
It doesn’t seem excessive?

Thanks

4

Look at foreman-rake reports:expire and foreman-rake audits:expire
Those commands will clean reports and audit logs. For example you can delete all reports older than 1 day by executing ‘foreman-rake reports:expire days=1’ and same for audits. This can take quite a long time. Choose your period that you want to preserve reports and audits.

What about your amount of facts per host, I think that 115 facts for host is OK. In my Foreman setup many hosts have more than 200 facts. But they collected every hour when Puppet runs on host.

5

Thanks for your suggestions.
I’ve ran both commands with days=1, but the situation did not improve (tried service restart also).
The response times are now even worse, over 60s (maybe unrelated to foreman-rake expire).

It’s particularly painful, because the server doesn’t do any caching. Every time I want to see the hostlist, I have to wait 20-70s.

Not sure what else I could try…

Thanks

6

Meanwhile, I have updated to the latest katello 4.9.2
No difference.

7

Foreman 3.8 will include Bug #36715: Report Template output generation can take hours to complete if the template is only about printing different host facts - Foreman

If it’s facts that’s slowing you down, the associated fix Fixes #36715 - Speed up host fact retrieval when you only need a subset of facts by jeremylenz · Pull Request #9819 · theforeman/foreman · GitHub will probably help.

1 Like
8

Thanks, this looks very promising!
But I don’t know if it’s the host facts that are causing the performance issue, or something else.
Not sure how to figure out by now.

Do you know why there is no caching?

Thanks