Problem: Listing and listing job invocations on Foreman gui as non-admin user is painfully slow (20-60 seconds).
Observed Endpoints:
/job_invocations
admin: 2-3 sec
non-admin: 4-6 sec
/job_invocations/[id]
admin: 3 sec
non-admin: 25-55 sec < Worst offender by far
template_invocations/[id]
admin: 3-4 sec
non-admin: 9 sec
Expected outcome: non-admin requests should be nearly as performant as a full admin
Foreman and Proxy versions: 1.23.0
Foreman and Proxy plugin versions: 1.23.0
Distribution and version: CentOS 7.7.1908
Other relevant data:
- CPU usage for passenger process goes to ~100% for the duration of the request (Passenger RackApp: /usr/share/foreman )
- only 105 hosts in the Foreman system
- 16GB RAM and 2 vCPU - low utilization normally. Never see the system have less than 3 GB free RAM.
This seems very similar to Bug #19742: API call for fact_values is slow when user is non-admin , Foreman 1.15.0 - Foreman, just with Job Invocations instead of facts.
Note: We also had to disable batching due to this bug, which I suspect may be compounding the issue: Remote Execution job runs the command on all servers simultaneously, even when set to distribute over X seconds
I have debug logging and permissions logging enabled. I couldn’t find anything useful except to note that the rendered requests really did take a very long time.
e.g.
Completed 200 OK in 32792ms (Views: 31735.4ms | ActiveRecord: 961.6ms)
Completed 200 OK in 55735ms (Views: 47333.0ms | ActiveRecord: 7576.2ms)
Completed 200 OK in 42509ms (Views: 37509.6ms | ActiveRecord: 4392.4ms)
Completed 200 OK in 23215ms (Views: 22198.0ms | ActiveRecord: 923.7ms)
Is anyone else noticing this?
What other info or troubleshooting would be helpful?
Cheers!