Hello all!
After upgrading to 1.13.1 today, I immediately noticed the huge delays in
search through the hosts in both WebUI and API (the latter is actually
worse than WebUI). Here are few examples:
time curl -kSs -u admin:$PASS https://localhost/api/hosts | jq '.' | grep
total
"total": 1695,
"subtotal": 1695,
real 0m0.816s
user 0m0.040s
sys 0m0.076s
But the search through my 1695 hosts on this instances takes 2.5 minutes
(!):
time curl -kSs -u admin:$PASS https://localhost/api/hosts?search=test |
jq '.' | grep total
"total": 1695,
"subtotal": 1,
real 2m25.147s
user 0m0.053s
sys 0m0.063s
For comparison, the same operations in 1.7.1 (from which I migrated one of
the instances) give these results:
$ time curl -kSs -u admin:$PASS https://localhost/api/v2/hosts | jq '.' |
grep total
"total": 2561,
"subtotal": 2561,
real 0m1.057s
user 0m0.014s
sys 0m0.007s
$ time curl -kSs -u admin:$PASS https://localhost/api/v2/hosts?search=test |
jq '.' | grep total
"total": 2561,
"subtotal": 1,
real 0m1.101s
user 0m0.012s
sys 0m0.007s
I did a bit of performance tuning on this instances as soon as all of those
clients started coming in hundreds, I started getting 500 errors
("webserver is under heavy load"). That helped me solve the 500 errors, but
not the search ones.
I also made sure there are no issues with accessing the DB out of Foreman,
but that's kind of redundant step anyway as there are no issues listing all
of the hosts, which wouldn't be the case if there were DB issues.
Any ideas on the next steps?
Thanks!
