High memory usage of Audits page with Foreman 3.8.0

Support
1

Problem: Browsing the audits page in the web ui causes a high memory usage in Puma

Expected outcome: Less memory consumption

Foreman and Proxy versions: 3.8.0

Foreman and Proxy plugin versions: 3.8.0

Distribution and version: AlmaLinux 8.8

Other relevant data:
Checking the audits page in the web ui in Monitor → Audits causes a high memory consumption for a Puma worker.
The page takes quite some time to load and during the page load, the memory consumption of the Puma worker increases up to ~1.3 GB.
See below for the output of foreman-puma-status of a freshly started Puma service, where I loaded the audit page multiple times:

# foreman-puma-status 
2291 (/usr/share/foreman/tmp/puma.state) Version: 6.3.1/ruby2.7.8p225 | Uptime: 19m13s | Phase: 0 | Load: 0[░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]30 | Req: 664
 └  2337 CPU:   0.0% Mem:  298 MB Uptime: 19m12s | Load: 0[░░░░░]5 | Req: 115
 └  2339 CPU:   0.0% Mem: 1331 MB Uptime: 19m12s | Load: 0[░░░░░]5 | Req: 92
 └  2340 CPU:   0.0% Mem: 1638 MB Uptime: 19m12s | Load: 0[░░░░░]5 | Req: 170
 └  2343 CPU:   0.0% Mem: 1331 MB Uptime: 19m12s | Load: 0[░░░░░]5 | Req: 51
 └  2344 CPU:   0.0% Mem:  295 MB Uptime: 19m12s | Load: 0[░░░░░]5 | Req: 109
 └  2348 CPU:   0.0% Mem: 1331 MB Uptime: 19m12s | Load: 0[░░░░░]5 | Req: 127

If the page is reloaded and the same worker process handles the request, the memory consumption grows further. I managed to hit >= 5GB of memory usage during debugging.

The /audits endpoint has also a very high allocation count:

2023-11-06T07:53:07 [I|app|2168c89f] Started GET "/audits?page=1&search=" for 131.169.185.191 at 2023-11-06 07:53:07 +0100
2023-11-06T07:53:07 [I|app|2168c89f] Processing by AuditsController#index as JSON
2023-11-06T07:53:07 [I|app|2168c89f]   Parameters: {"page"=>"1", "search"=>""}
2023-11-06T07:54:29 [I|app|2168c89f] Completed 200 OK in 82280ms (Views: 62.6ms | ActiveRecord: 25359.4ms | Allocations: 32488729)

The complete audit table seems to be pulled into memory on every request. Our audits table is ~1.3 GB and contains ~2 mio. entries, even with regular removal of old audit entries:

foreman=> select count(id) from audits;
  count  
---------
 2068873
(1 row)

foreman=> SELECT pg_size_pretty( pg_total_relation_size('audits') );
 pg_size_pretty 
----------------
 1316 MB
(1 row)

I can currently not indicate, which Foreman version introduced this issue.

2

Not sure if this is a problem of a specific version or it simply to much events for a good handling. Not sure if you are aware of the cleanup tasks you can use to remove old entries and if this is possible with your company’s policy, but you can find documentation on this topic at Administering Foreman

3

I am already expiring old audit records, currently I keep 180 days.
Our audit table is rather big due to a workflow polling the API with a personal access token every 5 seconds.
Every use of the personal access token results into a new audit entry to update the Last used at timestamp. That alone are just 17k entries every single day :frowning:

Coming back to the original issue, I tried to narrow down the issue between Foreman 2.5.4 and 3.8.0 in my staging environment. You seem to be right, too many events to handle.
All versions seem to be susceptible to this issue, even 2.5.4.

4

This sounds like there is an enhancement needed, either to allow suppressing such events or allowing to delete with a filter. While the first one would remove the load directly it would need another setting and it would only affect new entries so it could be to late for some cases. So I would prefer the second one, as the task cleanup already has a similar function and also the report cleanup can filter uneventful. Do you mind opening an issue at Issues - Foreman as I did not find a matching one already?