Hi there ! Problem:
I’m publishing a content view containing 48 repository (80 000 packages) every day without problem.
However after 6 or 7 days, I encounter a crash due to the system being seemingly out of memory.
Java invokes the oom-killer which kill mongodb when it’s publishing the CV
Doing a “katello-service restart” fixes the issue for a few days (a few publish) then it crashes again.
The system has 20 GB of RAM and 4 GB of SWAP, however, I do see a continuous increase of the Ram and swap usage, up until the swap is full and then the crash when the swap is full and a publish is triggered
Transparent Huge Page are disabled which helped a bit.
Adding 4GB of RAM helped a bit too but the problem came back
Having a reliable Publish that doesn’t crash after 7 ou 8 publish
Distribution and version:
CentOS Linux release 7.6.1810 (Core)
Other relevant data:
Do you have any “memory optimization trick” for Foreman ? I looked around, but besides the Transparent Huge Page setting, I didn’t find much.
Would increasing the Swap help ?
I already increased the ram as stated above but it just pushed the problem further.
/var/log/message log :
Dec 17 04:13:48 master-repo kernel: [1269272.647638] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 17 04:13:48 master-repo kernel: [1269272.647838] Swap cache stats: add 4168599, delete 4089346, find 17293084/17547978
Dec 17 04:13:48 master-repo kernel: [1269272.647839] Free swap = 0kB
Dec 17 04:13:48 master-repo kernel: [1269272.647840] Total swap = 4194300kB
Dec 17 04:13:49 master-repo kernel: [1269272.648170] Out of memory: Kill process 6648 (mongod) score 312 or sacrifice child
Dec 17 04:13:49 master-repo kernel: [1269272.648372] Killed process 6648 (mongod) total-vm:10724776kB, anon-rss:6854016kB, file-rss:0kB, shmem-rss:0kB
I have pulled this from our Red Hat Website, not sure if you can see this link so I will copy the info just incase but can you try this:
There are three options how to relief the memory pressure:
Usually, there is some another process consuming more than epxected amount of memory. Or a group of processes, when one e.g. sets passenger max pool size too high, causing many Passenger RackApp: /usr/share/foreman processes to pop up. We recommend checking this at the first place.
More memory might be needed for the Satellite server. This depends on the currently allocated RAM and Satellite usage.
One can limit wiredTiger max cache. Be aware that this can negatively affect Satellite performance . Red Hat has not tested impact of this option to performance. Please open a support case before setting this value. Two options of setting this permanently are possible:
By updating sysconfig file:
Edit the line in /etc/opt/rh/rh-mongodb34/sysconfig/mongod by adding the --wiredTigerCacheSizeGB 8 . Note that the particular value has not been tested and should be derived from a typical usage of your Satellite, rather :
Awesome, thanks you for your reply.
I can indeed see Red Hat KB but that way other people will have the info to, thank you for your paste.
Just for my personnal curiosity, the KB is show as “updated” the 10th of December, is it also its creation date ? Because I’m pretty sure I’ve looked for something a week or too ago and didn’t find anything.
Thank you again, I’ll try this solution and report back.
Thanks for this! I previously tried to set the cache size in mongod.conf, but I think puppet overwrote it. The default for mongod is to use something like 1/2 of system RAM, which is simply bonkers. I wonder if a more sane default when having an all-in-one Foreman install could be set?
I am also hitting the dynflowd memory leak… In my 16GB system, I dropped dynflowd to use 1GB max and then it recycles itself: