Loading Job Invocations extremely slow for non-admin users

Problem: Listing and listing job invocations on Foreman gui as non-admin user is painfully slow (20-60 seconds).

Observed Endpoints:
admin: 2-3 sec
non-admin: 4-6 sec

admin: 3 sec
non-admin: 25-55 sec < Worst offender by far

admin: 3-4 sec
non-admin: 9 sec

Expected outcome: non-admin requests should be nearly as performant as a full admin

Foreman and Proxy versions: 1.23.0

Foreman and Proxy plugin versions: 1.23.0

Distribution and version: CentOS 7.7.1908

Other relevant data:

  • CPU usage for passenger process goes to ~100% for the duration of the request (Passenger RackApp: /usr/share/foreman )
  • only 105 hosts in the Foreman system
  • 16GB RAM and 2 vCPU - low utilization normally. Never see the system have less than 3 GB free RAM.

This seems very similar to Bug #19742: API call for fact_values is slow when user is non-admin , Foreman 1.15.0 - Foreman, just with Job Invocations instead of facts.

Note: We also had to disable batching due to this bug, which I suspect may be compounding the issue: Remote Execution job runs the command on all servers simultaneously, even when set to distribute over X seconds

I have debug logging and permissions logging enabled. I couldn’t find anything useful except to note that the rendered requests really did take a very long time.

Completed 200 OK in 32792ms (Views: 31735.4ms | ActiveRecord: 961.6ms)
Completed 200 OK in 55735ms (Views: 47333.0ms | ActiveRecord: 7576.2ms)
Completed 200 OK in 42509ms (Views: 37509.6ms | ActiveRecord: 4392.4ms)
Completed 200 OK in 23215ms (Views: 22198.0ms | ActiveRecord: 923.7ms)

Is anyone else noticing this?
What other info or troubleshooting would be helpful?


So do we think this is a bug and I should create an official “issue” on the bug tracker? or

Yes, it may help to attach production.log with sql logger enabled and debug log level set to debug.

I created issue: Bug #28723: Loading Job and Template Invocations consumes massive RAM and extremely slow for non-admin users - Foreman Remote Execution - Foreman

This issue was getting worse and worse. This morning it took 20 minutes+ to display the Job Invocations page for my NOC.

I brought in a Ruby guy to help troubleshoot… After enabling debug logging, we found one template was taking 30 seconds. He commented out a single bit that renders Host Actions. This restored performance, at the expense of losing Host Actions buttons.

**/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.8.2/app/views/job_invocations commented out partial template called **

** <%= render ‘host_actions_td’, options %>**