High memory usage after upgrading to 3.1.3

[heyitsgary]

Problem: Foreman server runs out of memory

Expected outcome: No memory issue

Foreman and Proxy versions: Foreman 3.1.3

Foreman and Proxy plugin versions:
foreman-tasks 5.2.3
foreman_ansible 7.0.4
foreman_bootdisk 19.0.3
foreman_cockpit 2.0.3
foreman_discovery 19.0.1
foreman_docker 5.0.0
foreman_hooks 0.3.17
foreman_opensca 5.1.1
foreman_puppet 2.0.7
foreman_remote_execution 5.0.6
foreman_snapshot_management 2.0.1
katello 4.3.1

Distribution and version:
Centos 7
Other relevant data:
I went through the process to upgrade my Foreman servers to 3.1 from 2.5.x
I went to 3.0 to 3.1 and am currently on 3.1.3.
After the successful upgrade I see that the amount of ram and swap is nearly full now, I cannot find out why.

[ipa_gluciano@decdc01plcfg001 ~]$ sudo foreman-puma-status
103438 (/usr/share/foreman/tmp/puma.state) Uptime: 1d 0h | Phase: 0 | Load: 2[██░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]90 | Req: 141200
└ 104996 CPU: 0.0% Mem: 640 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 7011
└ 105001 CPU: 0.0% Mem: 743 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 14431
└ 105008 CPU: 0.0% Mem: 639 MB Uptime: 1d 0h | Load: 1[█░░░░]5 | Req: 7890
└ 105014 CPU: 0.0% Mem: 903 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 3254
└ 105020 CPU: 0.0% Mem: 659 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 6896
└ 105025 CPU: 0.0% Mem: 672 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 9540
└ 105034 CPU: 0.0% Mem: 643 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 7912
└ 105037 CPU: 0.0% Mem: 871 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 12101
└ 105042 CPU: 0.0% Mem: 694 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 10496
└ 105053 CPU: 0.0% Mem: 648 MB Uptime: 1d 0h | Load: 1[█░░░░]5 | Req: 5204
└ 105060 CPU: 0.0% Mem: 673 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 11795
└ 105063 CPU: 0.0% Mem: 684 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 4122
└ 105079 CPU: 0.0% Mem: 854 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 6377
└ 105091 CPU: 0.0% Mem: 734 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 6273
└ 105095 CPU: 0.0% Mem: 639 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 3943
└ 105131 CPU: 0.0% Mem: 657 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 5350
└ 105158 CPU: 0.0% Mem: 740 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 7777
└ 105185 CPU: 0.0% Mem: 845 MB Uptime: 1d 0h | Load: 0[░░░░░]5 | Req: 10828

Are there any steps that I can take to resolve this issue? Thank you

[dralley]

700mb per Puma workers is… that’s a lot of memory.

[heyitsgary]

Any suggestions on what I can do from here?

[iballou]

Hi @heyitsgary,

I found some good debugging steps from another post. Take a look at these two messages and consider trying what they’re suggesting. Then, we can go from there:

Puma cluster worker using huge amounts of memory

Can you tell from production.log what are the controllers that are being used? What’s the workload? If you are running a monitoring system, ideally, can you tell when a process saw a spike so we can correlate this with logs in the production.log? We really need to narrow this down a bit, Foreman as many other Ruby apps do have memory leaks, we are pretty successful with finding them and fixing them. But there must be something special with your setup, 3.1 is going through heavy testing at the m…

Puma cluster worker using huge amounts of memory

Cheap trick to find out which controller/action was slow as the first (typically when there is some problem with e.g. accidentally loading all records into the DB the offender is also very slow) is to restart, tail production.log and grep records with NNNNNms (5+ digits milliseconds). There are typically the offenders consuming all the memory.

[heyitsgary]

Hi @iballou

Thanks for linking those. I did find them a few days ago and tried those rake commands to clear any orphans. That did not seem to resolve the mem issue. I did find this tuning guide and looked more into the Puma tuning
https://docs.theforeman.org/nightly/Performance_Tuning_Guide/index-foreman-el.html#Puma_Tunings_performance-tuning

I dropped the amount of Puma workers and that so far has seemed to help quite a bit. Still monitoring the servers and will make adjustments but I think this is a very good start.

[iballou]

Glad to hear dropping Puma workers helped. The posts I linked also mention “tail production.log and grep records with NNNNNms (5+ digits milliseconds)” after a server restart. That could help find if there are certain actions that are eating up resources.