Performance Problems on Webinterface /puppetclasses

Support
1

Hi,

we have a huge performance problem on the puppet class view. It takes
abount 30 seconds to open the page.

2016-10-13 08:21:47 4d13dccb [app] [I] Started GET "/puppetclasses" for
172.29.70.9 at 2016-10-13 08:21:47 +0200
2016-10-13 08:21:47 4d13dccb [app] [I] Processing by
PuppetclassesController#index as HTML
2016-10-13 08:21:47 4d13dccb [app] [D] Cache read:
_session_id:4d13dccb22fd168676dfa37aa39b7057
2016-10-13 08:21:47 4d13dccb [app] [D] Setting current user thread-local
variable to mtnu
2016-10-13 08:21:47 4d13dccb [app] [D] Cache read:
authorize_login_delegation
2016-10-13 08:21:47 4d13dccb [app] [D] Cache read:
authorize_login_delegation
2016-10-13 08:21:47 4d13dccb [app] [D] Cache read: idle_timeout
2016-10-13 08:21:47 4d13dccb [app] [D] Setting current organization
thread-local variable to zed
2016-10-13 08:21:47 4d13dccb [app] [D] Cache read: entries_per_page
2016-10-13 08:22:16 4d13dccb [app] [I] Rendered
puppetclasses/index.html.erb within layouts/application (909.7ms)
2016-10-13 08:22:16 4d13dccb [app] [I] Rendered
common/_searchbar.html.erb (5.5ms)
2016-10-13 08:22:16 4d13dccb [app] [I] Rendered
layouts/_application_content.html.erb (6.0ms)
2016-10-13 08:22:16 4d13dccb [app] [D] Cache read: use_gravatar
2016-10-13 08:22:16 4d13dccb [app] [I] Rendered
home/_user_dropdown.html.erb (2.8ms)
2016-10-13 08:22:16 4d13dccb [app] [D] Cache read:
views/tabs_and_title_records-8 ({:skip_digest=>true})
2016-10-13 08:22:16 4d13dccb [app] [I] Read fragment
views/tabs_and_title_records-8 (0.3ms)
2016-10-13 08:22:16 4d13dccb [app] [I] Rendered home/_topbar.html.erb
(7.1ms)
2016-10-13 08:22:16 4d13dccb [app] [I] Rendered layouts/base.html.erb
(8.9ms)
2016-10-13 08:22:16 4d13dccb [app] [I] Completed 200 OK in 28831ms (Views:
901.8ms | ActiveRecord: 4243.4ms)
2016-10-13 08:22:16 4d13dccb [app] [D] Cache write:
_session_id:4d13dccb22fd168676dfa37aa39b7057 ({:expires_in=>86400})

We have 289 Puppetclasses importet in foreman. We are using the
organization feature, so users can only manage the puppetclasses that they
have importet on the smart proxy which belongs to their organization. This
is filtered via user groups and permissions. The permission for viewing the
classes is "environment ~ $org_name". (The environments of the
organizations beginns with the name of the organization.)

If we search for a puppetclass that only one organization use, the search
takes only about 10 secounds. But if we search for a class almost all
organizations use, the search takes several minutes and the foreman
processes eats almost all ram on the server.

The log above shows that Views and DB are quick, but the complete request
takes 28831ms.

Is there a way we can debug this? We are using Foreman 1.13.0

Thanks for your help!

Martin

2

On the same note - after upgrading to 1.13.1 today, I immediately noticed
the huge delays in search through the hosts in both WebUI and API (the
latter is actually worse than WebUI). Here are few examples:

time curl -kSs -u admin:$PASS https://localhost/api/hosts | jq '.' | grep

total
"total": 1695,
"subtotal": 1695,

real 0m0.816s
user 0m0.040s
sys 0m0.076s

But the search through my 1695 hosts on this instances takes 2.5 minutes
(!):

time curl -kSs -u admin:$PASS https://localhost/api/hosts?search=test |

jq '.' | grep total
"total": 1695,
"subtotal": 1,

real 2m25.147s
user 0m0.053s
sys 0m0.063s

For comparison, the same operations in 1.7.1 (from which I migrated one of
the instances) give these results:

$ time curl -kSs -u korekhov:$PASS https://localhost/api/v2/hosts | jq '.'
> grep total
"total": 2561,
"subtotal": 2561,

real 0m1.057s
user 0m0.014s
sys 0m0.007s

$ time curl -kSs -u korekhov:$PASS
https://localhost/api/v2/hosts?search=test | jq '.' | grep total
"total": 2561,
"subtotal": 1,

real 0m1.101s
user 0m0.012s
sys 0m0.007s

Would much appreciate the help figuring this out.

3

Probably need to start from Apache/Passenger tuning as I started to get
these on puppet runs:

Could not retrieve catalog from remote server:
Error 503 on SERVER: <h1>This website is under heavy load</h1><p>We're
sorry, too many people are accessing this website at the same time. We're
working on this problem. Please try again later.</p>

4

Passenger tuning only helped with puppet runs, but not search problem.