Foreman 3.16 memory issues (OOM)

robb · November 28, 2025, 11:29am

Problem:

We are seeing regular OOM errors on our Foreman/Katello server. The server has 90GB RAM which I think should be more than enough.

The memory issues cause the system to hang and become unavailable.

Expected outcome:

The system to not consume all the hosts memory and remain stable.

Foreman and Proxy versions:

foreman-3.16.1-1

katello-4.18.1-1

Installed Packages

Foreman and Proxy plugin versions:

Distribution and version:

AlmaLinux release 9.7 (Moss Jungle Cat)

5.14.0-611.5.1.el9_7.x86_64

Other relevant data:

An example of the logs from one of the OOMs events.

Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel:
Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel:
Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Free swap Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: [ pid ] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 610] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 625] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 791] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 826] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 830] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 831] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 833] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 834] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 835] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 838] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 867] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 868] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 912] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 933] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 951] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 976] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 977] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 979] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1009] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1023] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1027] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1064] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1081] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1082] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1090] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1286] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1398] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1400] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1404] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1405] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1406] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1407] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1408] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1409] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1441] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1460] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1461] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1518] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1542] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 1748] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2630] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2631] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2632] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2633] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2634] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2635] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2636] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2637] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2641] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 2735] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3078] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3079] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3080] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3458] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3462] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3463] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3472] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3473] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3474] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3488] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3493] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3497] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3501] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3502] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3506] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3510] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3521] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3525] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 3620] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 5574] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 19116] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 19117] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 24545] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 60253] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 82692] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 82846] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 82847] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 82964] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 399822] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 629055] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 744597] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 773342] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1011528] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1235742] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1235745] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1235747] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1235749] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1235751] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1402873] Nov 28 09:24:28 uk1-foremansvr1 kernel: [1948521] Nov 28 09:24:28 uk1-foremansvr1 kernel: [2292207] Nov 28 09:24:28 uk1-foremansvr1 kernel: [2538915] Nov 28 09:24:28 uk1-foremansvr1 kernel: [2538922] Nov 28 09:24:28 uk1-foremansvr1 kernel: [2634562] Nov 28 09:24:28 uk1-foremansvr1 kernel: [2634563] Nov 28 09:24:28 uk1-foremansvr1 kernel: [2798233] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3277696] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3277705] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3277707] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3277780] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3277784] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3277790] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3707404] Nov 28 09:24:28 uk1-foremansvr1 kernel: [3707678] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052377] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052379] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052384] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052385] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052386] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052395] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052399] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052400] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052401] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052405] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052407] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052410] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052412] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052414] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052416] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052425] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052429] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052434] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052435] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052437] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052438] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4052441] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4137798] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4137802] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4137804] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4137811] Nov 28 09:24:28 uk1-foremansvr1 kernel: [4137812] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 139751] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 140200] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 181416] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217233] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217530] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217540] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217542] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217544] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217901] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217907] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217910] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217911] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217913] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217932] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217933] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217937] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217944] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217952] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217960] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217964] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 217965] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 230644] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 259324] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 281972] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 286739] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 286740] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 286741] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 286742] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 286743] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 286755] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 286936] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287229] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287237] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287244] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287249] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287250] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287314] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287318] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287319] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 287320] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 288156] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 288626] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 290705] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 290706] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 290707] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 293636] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 315228] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 316737] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 317109] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 318161] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 319129] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 319130] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 319577] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 319578] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 319579] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 319580] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321474] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321475] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321498] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321505] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321517] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321534] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321545] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321555] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321558] Nov 28 09:24:28 uk1-foremansvr1 kernel: [ 321560] Nov 28 09:24:28 uk1-foremansvr1 kernel: Nov 28 09:24:28 uk1-foremansvr1 kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014
Call Trace:
dump_stack_lvl+0x34/0x48
dump_header+0x49/0x212
oom_kill_process.cold+0xb/0x10
out_of_memory+0xee/0x2b0
__alloc_pages_slowpath.constprop.0+0x793/0xb10
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1e0
? filemap_get_entry+0xe3/0x140
folio_alloc+0x12/0x30
__filemap_get_folio+0x189/0x2e0
filemap_fault+0x4d5/0x8b0
__do_fault+0x31/0x150
do_read_fault+0x11e/0x1d0
do_pte_missing+0x157/0x200
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x215/0x620
exc_page_fault+0x61/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x55d2dfc84e10
Code: Unable to access opcode bytes at 0x55d2dfc84de6.
RSP: 002b:00007ffc1b004438 EFLAGS: 00010246
RAX: 0000000000000006 RBX: 000055d2e11eb350 RCX: 000055d2e24f4300
RDX: 000055d2e280defc RSI: 0000000000000006 RDI: 0000000000000006
RBP: 00007ffc1b004450 R08: 0000000000000081 R09: 000055d2e11eb350
R10: 000055d2e24fc460 R11: 0000000000000246 R12: 000055d2e25deeb8
R13: 00007ffc1b004548 R14: 00007ffc1b004580 R15: 41535c5700000002
Mem-Info:
active_anon:13506418 inactive_anon:10605264 isolated_anon:0#012 active_file:8733 inactive_file:12660 isolated_file:0#012 unevictable:0 dirty:0 writeback:5426#012 slab _reclaimable:41590 slab_unreclaimable:41023#012 mapped:440981 shmem:942003 pagetables:84312#012 sec_pagetables:0 bounce:0#012 kernel_misc_reclaimable:0#012 free:142454 free_pcp:0 free_cma:0
Node 0 active_anon:54025672kB inactive_anon:42421056kB active_file:35288kB inactive_file:50284kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1763924k B dirty:0kB writeback:21704kB shmem:3768012kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:22512kB pagetables:337248kB sec_pagetables:0kB all_unreclaimable? yes
Node 0 DMA free:11264kB boost:0kB min:4kB low:16kB high:28kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0 kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve: 0 2479 96026 96026 96026
Node 0 DMA32 free:374736kB boost:0kB min:1004kB low:3504kB high:6004kB reserved_highatomic:0KB active_anon:431940kB inactive_anon:1716960kB active_file:520kB inactive _file:3632kB unevictable:0kB writepending:764kB present:3129176kB managed:2539352kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve: 0 0 93546 93546 93546
Node 0 Normal free:183816kB boost:0kB min:38592kB low:134376kB high:230160kB reserved_highatomic:145408KB active_anon:53594000kB inactive_anon:40703828kB active_file: 35760kB inactive_file:45660kB unevictable:0kB writepending:20188kB present:97517568kB managed:95791924kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve: 0 0 0 0 0
Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
Node 0 DMA32: 2734kB (UM) 3008kB (UME) 22016kB (UE) 17832kB (UE) 5464kB (UME) 39128kB (UME) 11256kB (UE) 8512kB (UE) 21024kB (UE) 42048kB (UME) 824096kB (U ME) = 374180kB
Node 0 Normal: 27084kB (UME) 17868kB (UME) 45016kB (UME) 21232kB (UME) 8164kB (UME) 125128kB (UME) 52256kB (UME) 18512kB (UME) 111024kB (UME) 42048kB (UME) 20*4096kB (ME) = 184192kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
2084078 total pagecache pages
1120694 pages in swap cache
= 236kB
Total swap = 8388604kB
25165684 pages RAM
0 pages HighMem/MovableOnly
579025 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Tasks state (memory values in pages):
uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
0 610 49303 31690 434176 224 -250 systemd-journal
0 625 9628 48 90112 832 -1000 systemd-udevd
790] 32 790 3689 96 65536 192 0 rpcbind
0 791 5795 72 65536 224 -1000 auditd
824] 81 824 2732 132 61440 160 -900 dbus-broker-lau
825] 81 825 1424 187 49152 64 -900 dbus-broker
0 826 65044 320 135168 480 0 NetworkManager
0 830 604275 1815 294912 477 0 ir_agent
0 831 20693 64 61440 32 0 irqbalance
832] 996 832 679 96 45056 32 0 lsmd
0 833 708 32 40960 32 0 mcelog
0 834 24433 128 69632 32 0 qemu-ga
2 835 5011 128 73728 224 0 rngd
0 838 5705 437 86016 352 0 systemd-logind
849] 991 849 21215 13 73728 128 0 chronyd
0 867 364438 1483 225280 224 0 rapid7_agent_co
0 868 530561 1204 249856 224 0 rapid7_endpoint
0 912 2145 320 73728 128 0 check_mk_agent
918] 982 918 4151 393 69632 160 0 cmk-agent-ctl
921] 985 921 57556 12904 385024 8415 0 smart-proxy
0 933 2479 96 53248 64 0 oddjobd
942] 984 942 46697 1501 409600 30560 0 pulpcore-api
947] 984 947 46411 2012 413696 29792 0 pulpcore-conten
0 951 4938 160 77824 384 -1000 sshd
969] 91 969 2578366 321207 4743168 150299 0 java
0 976 66624 276 159744 3264 0 tuned
0 977 21316 320 163840 512 0 winbindd
0 979 14150 17 86016 224 0 gssproxy
985] 984 985 46393 3237 409600 28480 0 pulpcore-worker
990] 984 990 46392 3499 413696 28288 0 pulpcore-worker
992] 984 992 46392 3671 409600 28192 0 pulpcore-worker
995] 984 995 46394 3378 417792 28384 0 pulpcore-worker
999] 984 999 46392 2992 413696 28832 0 pulpcore-worker
1002] 984 1002 46393 3704 409600 28128 0 pulpcore-worker
1004] 984 1004 46392 3141 413696 28704 0 pulpcore-worker
1005] 984 1005 46392 2696 409600 29120 0 pulpcore-worker
0 1009 54346 11109 335872 7072 0 puppet
1011] 983 1011 1471776 224503 10358784 972768 0 redis-server
0 1023 132857 5429 434176 205 0 rsyslogd
26 1027 1107313 18240 417792 416 -1000 postmaster
0 1064 6009 453 90112 448 0 httpd
0 1081 3287 160 61440 192 0 atd
0 1082 2147 192 57344 160 0 crond
0 1090 764 64 45056 0 0 agetty
26 1286 17541 435 106496 416 0 postmaster
0 1398 22791 160 172032 608 0 smbd
0 1400 21286 239 139264 544 0 wb[UK1-FOREMANS
26 1404 1107429 356459 7417856 384 0 postmaster
26 1405 1107378 13483 294912 416 0 postmaster
26 1406 1107313 4267 151552 416 0 postmaster
26 1407 1107658 459 167936 416 0 postmaster
26 1408 17031 299 98304 416 0 postmaster
26 1409 1107582 427 147456 448 0 postmaster
0 1441 21795 239 163840 672 0 wb[ION]
0 1460 21751 173 122880 640 0 smbd-notifyd
0 1461 21755 173 126976 640 0 smbd-cleanupd
1506] 998 1506 746133 441 258048 768 0 polkitd
0 1518 21627 310 151552 544 0 wb-idmap
0 1542 11676 196 135168 4736 0 ir_agent
0 1748 59842 288 393216 2880 0 linux_agent_exe
26 2630 1108756 13323 499712 544 0 postmaster
26 2631 1108701 12875 491520 512 0 postmaster
26 2632 1108756 13387 503808 544 0 postmaster
26 2633 1108708 12779 491520 512 0 postmaster
26 2634 1108708 12747 491520 544 0 postmaster
26 2635 1108701 11627 462848 576 0 postmaster
26 2636 1108708 12619 487424 576 0 postmaster
26 2637 1108738 13163 483328 480 0 postmaster
2638] 984 2638 83606 29395 425984 3008 0 pulpcore-conten
26 2641 1108862 14293 462848 544 0 postmaster
26 2735 1108716 3563 245760 736 0 postmaster
2994] 986 2994 241314 203388 1986560 17817 0 sidekiq
2997] 986 2997 295081 239473 2326528 23809 0 sidekiq
2998] 986 2998 352592 313900 2842624 16517 0 sidekiq
3000] 986 3000 199177 155395 1576960 16909 0 sidekiq
3013] 986 3013 170122 83870 1339392 53871 0 sidekiq
26 3078 1108779 7179 389120 544 0 postmaster
26 3079 1108772 10379 458752 576 0 postmaster
26 3080 1108760 8779 401408 512 0 postmaster
26 3458 1108470 3467 253952 640 0 postmaster
26 3462 1108733 32203 2248704 544 0 postmaster
26 3463 1108780 32698 2273280 608 0 postmaster
26 3472 1108725 33707 2314240 575 0 postmaster
26 3473 1108720 33195 2273280 576 0 postmaster
26 3474 1108762 33945 2252800 576 0 postmaster
26 3488 1108479 4075 258048 640 0 postmaster
26 3493 1108719 35243 2416640 640 0 postmaster
26 3497 1108471 3595 253952 608 0 postmaster
26 3501 1108736 35207 2428928 544 0 postmaster
26 3502 1108740 34315 2420736 544 0 postmaster
26 3506 1108704 45685 3805184 576 0 postmaster
26 3510 1108444 2315 237568 576 0 postmaster
26 3521 1108479 4011 258048 672 0 postmaster
26 3525 1108690 42059 3383296 640 0 postmaster
26 3620 1108684 46329 3715072 640 0 postmaster
26 5574 1108732 34316 2400256 544 0 postmaster
26 19116 1108731 34380 2396160 608 0 postmaster
26 19117 1108741 34092 2387968 576 0 postmaster
26 24545 1108729 34732 2379776 544 0 postmaster
26 60253 1108751 34476 2367488 512 0 postmaster
26 82692 1108677 45256 3645440 556 0 postmaster
26 82846 1108763 34700 2392064 620 0 postmaster
26 82847 1108764 33132 2318336 588 0 postmaster
26 82964 1108763 35052 2396160 556 0 postmaster
26 399822 1108700 45774 3653632 554 0 postmaster
0 629055 112326 381 212992 7840 0 ir_agent
0 744597 11676 236 131072 4736 0 ir_agent
26 773342 1108733 31374 1794048 554 0 postmaster
0 1011528 145094 267 221184 7840 0 ir_agent
0 1235742 26857 948 225280 2848 0 s1-orchestrator
980 1235745 330572 4847 434176 5376 0 s1-network
0 1235747 80392 12584 442368 15008 0 s1-scanner
0 1235749 583654 92704 1839104 22368 0 s1-agent
0 1235751 47871 1408 163840 2880 0 s1-firewall
26 1402873 1108636 36526 2625536 522 0 postmaster
0 1948521 11676 1126 131072 3872 0 ir_agent
0 2292207 145094 1169 221184 6912 0 ir_agent
0 2538915 953062 11078 610304 2784 0 ir_agent
0 2538922 11676 2277 131072 2720 0 ir_agent
26 2634562 1108745 8493 425984 395 0 postmaster
26 2634563 1108760 8301 438272 363 0 postmaster
0 2798233 145094 7157 225280 992 0 ir_agent
984 3277696 83621 8889 417792 23350 0 pulpcore-conten
984 3277705 83621 8889 417792 23318 0 pulpcore-conten
984 3277707 83621 8889 417792 23318 0 pulpcore-conten
26 3277780 1108864 13755 446464 331 0 postmaster
26 3277784 1108864 14479 450560 331 0 postmaster
26 3277790 1108864 14047 454656 331 0 postmaster
984 3707404 83621 8854 421888 23385 0 pulpcore-conten
26 3707678 1108863 15754 458752 363 0 postmaster
984 4052377 83621 8854 417792 23321 0 pulpcore-conten
984 4052379 83621 8822 413696 23321 0 pulpcore-conten
26 4052384 1108864 14837 442368 331 0 postmaster
26 4052385 1108864 14726 454656 363 0 postmaster
984 4052386 83621 8854 417792 23385 0 pulpcore-conten
984 4052395 83621 8895 413696 23321 0 pulpcore-conten
26 4052399 1108864 15398 462848 331 0 postmaster
984 4052400 102058 8822 425984 23353 0 pulpcore-conten
984 4052401 83621 8854 417792 23417 0 pulpcore-conten
26 4052405 1108864 14838 450560 331 0 postmaster
26 4052407 1108864 15566 458752 331 0 postmaster
984 4052410 83621 8834 413696 23331 0 pulpcore-conten
26 4052412 1108864 14926 462848 331 0 postmaster
984 4052414 83621 8822 417792 23385 0 pulpcore-conten
26 4052416 1108864 14411 454656 331 0 postmaster
26 4052425 1108864 15986 454656 331 0 postmaster
984 4052429 83621 8822 417792 23385 0 pulpcore-conten
984 4052434 83621 8854 413696 23353 0 pulpcore-conten
26 4052435 1108864 15941 462848 331 0 postmaster
984 4052437 83621 8911 417792 23385 0 pulpcore-conten
26 4052438 1108864 15908 458752 331 0 postmaster
26 4052441 1108864 16306 458752 331 0 postmaster
0 4137798 5439 672 94208 0 0 sshd
0 4137802 6257 990 90112 0 100 systemd
0 4137804 44726 1300 118784 645 100 (sd-pam)
0 4137811 5438 627 86016 64 0 sshd
0 4137812 2214 544 69632 0 0 bash
986 139751 129937 98216 950272 0 0 rails
26 140200 1108786 6196 286720 363 0 postmaster
0 181416 1382076 10755 417792 0 0 rapid7_velocira
986 217233 8774797 8723834 70238208 0 0 rails
26 217530 1108471 3405 253952 363 0 postmaster
26 217540 1108472 4205 307200 363 0 postmaster
26 217542 1108530 4557 331776 363 0 postmaster
26 217544 1108720 7362 311296 363 0 postmaster
986 217901 8752815 8668382 71086080 0 0 rails
26 217907 1108471 3373 253952 363 0 postmaster
26 217910 1108700 10061 598016 363 0 postmaster
26 217911 1108693 10221 593920 363 0 postmaster
26 217913 1108714 7363 311296 363 0 postmaster
984 217932 83621 8756 417792 23355 0 pulpcore-conten
26 217933 1108753 10476 557056 363 0 postmaster
26 217937 1108749 5518 278528 363 0 postmaster
26 217944 1108864 14345 471040 363 0 postmaster
986 217952 2686261 2620396 21569536 0 0 rails
26 217960 1108471 3373 253952 363 0 postmaster
26 217964 1108694 4589 335872 363 0 postmaster
26 217965 1108720 7394 311296 363 0 postmaster
26 230644 1110029 75962 2899968 363 0 postmaster
26 259324 1109362 61549 2416640 363 0 postmaster
26 281972 1109257 20427 1130496 363 0 postmaster
984 286739 46730 6984 393216 25166 0 pulpcore-api
984 286740 46730 6920 393216 25198 0 pulpcore-api
984 286741 46730 6984 393216 25134 0 pulpcore-api
984 286742 46730 6920 393216 25198 0 pulpcore-api
984 286743 46730 6920 393216 25230 0 pulpcore-api
0 286755 98962 2744 516096 2510 0 linux_agent_exe
986 286936 145070 120486 1183744 0 0 rails
26 287229 1108495 6765 360448 331 0 postmaster
26 287237 1108495 6701 360448 331 0 postmaster
26 287244 1108495 6797 360448 363 0 postmaster
26 287249 1108495 6765 360448 331 0 postmaster
26 287250 1108495 6765 360448 363 0 postmaster
26 287314 1108471 3373 253952 363 0 postmaster
26 287318 1108562 4301 315392 363 0 postmaster
26 287319 1108465 4141 311296 363 0 postmaster
26 287320 1108714 7395 311296 363 0 postmaster
0 288156 107388 7973 192512 0 0 ir_agent
0 288626 242798 8143 278528 0 0 ir_agent
26 290705 1109781 57533 2162688 331 0 postmaster
48 290706 616432 3355 372736 412 0 httpd
48 290707 616432 3579 372736 412 0 httpd
26 293636 1113804 76256 2580480 331 0 postmaster
26 315228 1109360 28720 1400832 363 0 postmaster
26 316737 1108947 20042 1011712 363 0 postmaster
26 317109 1110773 49192 2502656 363 0 postmaster
26 318161 1109133 22669 1056768 363 0 postmaster
26 319129 1110517 36885 2138112 363 0 postmaster
26 319130 1110507 31045 1814528 363 0 postmaster
26 319577 1109098 9812 544768 363 0 postmaster
26 319578 1109905 26572 1241088 363 0 postmaster
26 319579 1108655 6413 389120 395 0 postmaster
26 319580 1112991 40890 1597440 363 0 postmaster
26 321474 1110511 14043 786432 363 0 postmaster
26 321475 1110768 14344 761856 363 0 postmaster
26 321498 1108655 3885 253952 395 0 postmaster
26 321505 1108655 3917 253952 363 0 postmaster
0 321517 6009 315 81920 508 0 httpd
26 321534 1108455 1965 233472 395 0 postmaster
26 321545 1108610 2605 241664 395 0 postmaster
983 321555 1471810 248363 10342400 971238 0 redis-server
26 321558 1107635 589 180224 395 0 postmaster
0 321560 2145 219 65536 123 0 check_mk_agent
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/foreman.service,task=rails,pid=217233,uid=986
Out of memory: Killed process 217233 (rails) total-vm:35099188kB, anon-rss:34895336kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:68592kB oom_score_adj:0

robb · December 2, 2025, 11:19am

Please let me know if I can provide any more info - thanks in advance for any help!

quba42 · December 2, 2025, 11:58am

I guess we need to find out what actions are happening that are eating all the memory to the point where OOM kills them? In my experience memory problems are always caused by very specific tasks. For example large sync tasks where something has gone wrong. Or maybe publishing new content view versions with filters and dependency solving. Sometimes it is starting way to many things in parallel (although that should be fine unless you also have more workers than resources on the system).

robb · December 2, 2025, 1:12pm

Thanks for the response.

It happens multiple times a day - we generally only create content view versions once per month, so I don’t believe it is that.

We do have some scheduled jobs that run in AWX a couple of times a day that send data back to our Foreman server - I did wonder if it was anything to do with these jobs, but they don’t coincide with when the memory issues occur.

Perhaps these regular jobs gradually consume the memory, and the system is never able to properly release it again - the AWX jobs don’t run often enough to be the cause for each incident. You can see from the graph below how often it happens. Most of the sudden drops coincide with OOMs messages in the logs.

image779×213 29.6 KB

Regarding workers, we’ve tried various different combinations…does this seem ok for a 8 core host with 96GB RAM?

Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] Puma starting in cluster mode…
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Puma version: 6.6.1 (“Return to Forever”)
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Ruby version: ruby 3.0.7p220 (2024-04-23 revision 724a071175) [x86_64-linux]
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Min threads: 4
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Max threads: 8
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Environment: production
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Master PID: 2610451
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Workers: 4
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Restarts: (✔) hot (✖) phased (✖) refork
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Preloading application
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Activated unix:///run/foreman.sock
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] Use Ctrl-C to stop
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] Sending status to systemd every 1.0 sec
Dec 1 21:45:38 fmansvr foreman[2610451]: [2610451] * Starting control server on unix:///usr/share/foreman/tmp/sockets/pumactl.sock
Dec 1 21:45:39 fmansvr foreman[2610451]: [2610451] - Worker 0 (PID: 2610636) booted in 0.45s, phase: 0
Dec 1 21:45:39 fmansvr foreman[2610451]: [2610451] - Worker 2 (PID: 2610641) booted in 0.43s, phase: 0
Dec 1 21:45:39 fmansvr foreman[2610451]: [2610451] - Worker 1 (PID: 2610638) booted in 0.45s, phase: 0
Dec 1 21:45:39 fmansvr foreman[2610451]: [2610451] - Worker 3 (PID: 2610644) booted in 0.44s, phase: 0

robb · December 2, 2025, 2:19pm

Just some extra info - the system is currently almost out of memory.

# free -g
               total        used        free      shared  buff/cache   available
Mem:              93          91           1           4           6           2
Swap:              7           7           0

top - 14:18:39 up 11 days, 16 min,  1 user,  load average: 6.37, 33.37, 40.32
Tasks: 473 total,   2 running, 471 sleeping,   0 stopped,   0 zombie
%Cpu(s): 49.5 us,  5.7 sy,  0.0 ni, 38.1 id,  5.4 wa,  0.0 hi,  1.3 si,  0.0 st
MiB Mem :  96041.6 total,   1624.1 free,  93320.5 used,   6219.0 buff/cache
MiB Swap:   8192.0 total,      0.0 free,   8192.0 used.   2721.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2983810 foreman   20   0   41.5g  41.4g   7856 S  95.8  44.1 154:42.62 rails
2984282 foreman   20   0   15.4g  15.2g   8444 S  97.5  16.2  45:14.66 rails
2983802 foreman   20   0   21.2g  13.4g   8476 S   5.0  14.3  40:23.64 rails
3076046 redis     20   0 5887240   3.1g   2224 D  13.4   3.3   0:32.47 redis-server
   1404 postgres  20   0 4429864   2.6g   2.6g S   0.0   2.8  19:52.23 postmaster
3067968 postgres  20   0 4460524   1.9g   1.9g S   0.0   2.0   0:12.06 postmaster
3067986 postgres  20   0 4460516   1.9g   1.9g S   0.0   2.0   0:10.71 postmaster
   2997 foreman   20   0 1802076   1.5g   7840 S   0.0   1.6  57:04.56 sidekiq
3039635 postgres  20   0 4507276   1.4g   1.3g S   0.0   1.5   9:00.36 postmaster

The rails processes are quite large!

qcjames53 · December 2, 2025, 10:19pm

Like Quirin said, 95% and 97% CPU means this isn’t a memory leak. Foreman is definitely doing something to cause these memory spikes. I’m hoping we can help you out on the configuration side to trim down the required resources. Would you be able to send over what your foreman tasks page looks like during these periods of high activity (/foreman_tasks/tasks)? Just whatever tasks are in state: running. We’ll probably have some follow-up questions.

ikonia · December 3, 2025, 7:41pm

you should see in the logs OOM on the actions that are failing (that doesn’t mean they are the actions causing it though).

I normally find on OOM on the foreman/katello/satellite stack are caused by poor timing of resource management rather than resource management.

eg: a number of hosts checking at the same time, multiple repo syncs happening at the same time, a house keeping job running at the same time, all cumulate to an OOM error, where as the puppet runs on their own, fine, the repo syncs on their own fine etc.

I often finding adjusting the timings of workload puts the nodes back in working limits, however if the workloads are essential (eg: I had a client that required a puppet run every 15 minutes for compliance for a large fleet) breaking the functions out on to different nodes size right removed the problem

robb · December 8, 2025, 12:27pm

Hi,

We don’t have any repo syncs taking place, but there are some Foreman smart proxy syncs attempting to run.

The system is extremely sluggish right now, I see a constant stream in the Foreman production.log like this…

2025-12-08T12:18:33 [I|app|3d9acf72] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2025-12-08T12:18:33 [I|aud|450bf7e6] Nic::Managed (92233) update event on identifier on6853f84a180e4.1502, eno54.1502
2025-12-08T12:18:33 [I|aud|450bf7e6] Nic::Managed (92233) update event on attached_to on6853f84a180e4, eno54
2025-12-08T12:18:33 [I|app|60c668ab] Completed 304 Not Modified in 618ms (Views: 0.2ms | ActiveRecord: 402.0ms | Allocations: 345162)
2025-12-08T12:18:33 [I|app|53aa44c7] Rendered api/v2/taxonomies/index.json.rabl within api/v2/layouts/index_layout (Duration: 658.7ms | Allocations: 357214)
2025-12-08T12:18:33 [I|app|53aa44c7] Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 665.5ms | Allocations: 358877)
2025-12-08T12:18:33 [I|app|53aa44c7] Completed 200 OK in 1368ms (Views: 645.2ms | ActiveRecord: 66.0ms | Allocations: 701158)
2025-12-08T12:18:33 [I|app|a303f4b4] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2025-12-08T12:18:33 [I|app|a303f4b4] Parameters: {“id”=>“3349feb8-ff0b-4c03-b925-4c1d02f93c11”}
2025-12-08T12:18:33 [I|app|31fd75c6] Processing by Api::V2::HostgroupsController#index as JSON
2025-12-08T12:18:33 [I|app|31fd75c6] Parameters: {“per_page”=>“all”, “apiv”=>“v2”}
2025-12-08T12:18:33 [I|app|293541ad] Completed 200 OK in 657ms (Views: 2.6ms | ActiveRecord: 594.7ms | Allocations: 358036)
2025-12-08T12:18:33 [I|app|8fe077cb] Processing by Api::V2::UsersController#index as JSON
2025-12-08T12:18:33 [I|app|8fe077cb] Parameters: {“per_page”=>“all”, “apiv”=>“v2”}
2025-12-08T12:18:33 [I|app|3d9acf72] Completed 200 OK in 464ms (Views: 0.5ms | ActiveRecord: 24.2ms | Allocations: 285567)
2025-12-08T12:18:33 [I|app|96497ed9] Started GET “/rhsm/status” for 10.136.1.26 at 2025-12-08 12:18:33 +0000
2025-12-08T12:18:33 [I|app|f27e719d] Started GET “/rhsm/consumers/f5b9c2c8-fd4f-470f-99f2-2f5ba17e8bff/certificates/serials” for 10.136.136.180 at 2025-12-08 12:18:33 +0000
2025-12-08T12:18:33 [I|app|96497ed9] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2025-12-08T12:18:33 [I|app|f27e719d] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2025-12-08T12:18:33 [I|app|f27e719d] Parameters: {“id”=>“f5b9c2c8-fd4f-470f-99f2-2f5ba17e8bff”}
2025-12-08T12:18:33 [I|aud|450bf7e6] Nic::Managed (92146) update event on identifier on6853f84a180e4.1502, eno54.1502
2025-12-08T12:18:33 [I|aud|450bf7e6] Nic::Managed (92146) update event on attached_to on6853f84a180e4, eno54
2025-12-08T12:18:33 [I|app|950505d3] Completed 200 OK in 7127ms (Views: 0.7ms | ActiveRecord: 2813.0ms | Allocations: 1347689)
2025-12-08T12:18:34 [I|app|1b778b00] Started GET “/api/usergroups?per_page=all” for 10.136.52.124 at 2025-12-08 12:18:34 +0000
2025-12-08T12:18:34 [I|app|bdad5520] Started GET “/rhsm/consumers/d306fb3a-8fb2-400f-b92b-874a6744ff3e/content_overrides” for 10.136.136.180 at 2025-12-08 12:18:34 +0000
2025-12-08T12:18:34 [I|app|96497ed9] Completed 200 OK in 882ms (Views: 0.7ms | ActiveRecord: 26.8ms | Allocations: 167842)
2025-12-08T12:18:34 [I|app|a303f4b4] Completed 200 OK in 1500ms (Views: 0.7ms | ActiveRecord: 623.7ms | Allocations: 513416)
2025-12-08T12:18:34 [I|app|bdad5520] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2025-12-08T12:18:35 [I|app|bdad5520] Parameters: {“id”=>“d306fb3a-8fb2-400f-b92b-874a6744ff3e”}

This is the Tasks screen - it won’t load fully due to the performance issues…so not sure if its of any use. The running tasks are smart proxy syncs though.

I bumped up the RAM a bit as test to see if it would help…but the system just continues to grow and grow.

# free -g
               total        used        free      shared  buff/cache   available
Mem:             107         102           0           1           7           5
Swap:              7           0           7

And the usual suspects at the top…

top - 12:24:06 up  4:05,  1 user,  load average: 5.83, 5.52, 5.29
Tasks: 509 total,   1 running, 508 sleeping,   0 stopped,   0 zombie
%Cpu(s): 35.5 us,  0.6 sy,  0.0 ni, 63.0 id,  0.2 wa,  0.5 hi,  0.2 si,  0.0 st
MiB Mem : 110149.5 total,    957.0 free, 107199.2 used,   4760.2 buff/cache
MiB Swap:   8192.0 total,   7772.3 free,    419.7 used.   2950.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   3316 foreman   20   0   30.6g  30.4g  11812 S  99.0  28.3 174:25.30 rails
   3320 foreman   20   0   26.9g  26.7g  11664 S  99.7  24.8 121:55.76 rails
   3314 foreman   20   0   18.7g  18.5g  11752 S  98.3  17.2  85:21.81 rails
   3326 foreman   20   0   13.3g  12.9g  11888 S  98.7  12.0  73:56.98 rails
   1127 redis     20   0 6212736   4.7g   8852 S   0.7   4.3   2:28.57 redis-server
   1059 tomcat    20   0   11.8g   1.6g  29012 S   3.0   1.5  18:36.87 java
   1429 postgres  20   0 4429716 688876 686736 S   0.0   0.6   0:36.92 postmaster
   3302 foreman   20   0  696376 631548  15048 S   6.0   0.6  81:59.96 sidekiq

Thanks again for any help!

quba42 · December 8, 2025, 1:17pm

Those mainly perform work on the proxy, not on the main server. You are getting the OOM on your main server, right?

robb · December 8, 2025, 1:48pm

Correct, only on the Foreman server.

The smart proxy’s all seem pretty stable.

quba42 · December 8, 2025, 1:51pm

Then I am pretty sure the issue is not related to any running smart proxy syncs.

robb · December 9, 2025, 8:50am

Yep I don’t think so either - if anyone has any ideas that would be great, the system is often unusable and causing the team a lot of problems.

Perhaps there is some extra logging that can be enabled to find out what the rails processes are actually doing to use so much memory..?

Thanks!

quba42 · December 9, 2025, 12:37pm

I don’t think there is some logging trick that will tell us exactly what we need to know. Foreman is most likely “doing something” that is causing the memory leak, and you need to find out what that is. No one can figure it out from afar.

I presume you have tried things like re-starting the foreman services and looking what that does to the memory consumption? While you monitor what is happening with the memory, also monitor the production log, and whenever possible, what Foreman tasks are running. What happens after OOM kills the memory consuming process? Do you need to restart Foreman then?

iballou · December 10, 2025, 2:28pm

To add to the current advice, I would enable debug logging and monitor Foreman’s production.log.

This old post has some nice tips from lzap for looking into memory issues: Puma cluster worker using huge amounts of memory - #3 by jjvcuyler

In the log, with debug logging on, seeing what controllers are call frequently could give some direction.

Like:

2025-12-10T13:52:46 [I|app|55c3c877] Processing by Katello::Api::V2::HostPackagesController#containerfile_install_command as JSON
...
2025-12-10T13:52:46 [I|app|55c3c877] Completed 404 Not Found in 151ms (Views: 0.2ms | ActiveRecord: 23.6ms | Allocations: 50900)

Are there log records with large timing results? For example:

2022-01-25T05:14:55 [I|app|45ccd5b1] Completed 201 Created in 4909255ms (Views: 102.3ms | ActiveRecord: 352203.4ms | Allocations: 553340631)

These can help point to inefficient actions.

robb · December 11, 2025, 8:24am

Thanks - i’ll have a look through the other thread and enable debug logging…and hopefully report back soon!

robb · December 29, 2025, 11:53am

Just to update - we haven’t been able to find out what is causing the system to use up all the memory yet, but have applied a workaround to restart the rails processes if they start to use too much memory.

Not ideal, but its at least preventing the whole system to become unusable.

I’ll update when (or if) we find out what’s causing such high memory usage.