Memory crash kill mongod process

Problem:
Memory crash : I don’t know why, mongod crashes in out of memory yesterday evening. Any particular command was running.
I have 16Gb RAM and 2Gb of swap.I can increase swap but it’s already high.

Foreman and Proxy versions:
Foreman 2.0 and katello-3.15.0-0.5.rc2

Distribution and version:
RHEL 7.8

Other relevant data:

cf join logs
logs_mongod_outofmemory.log (88.2 KB)

There wasn’t a sync task going on when mongo crashed? If not, what sort of memory usage are you seeing on your machine when the server is idle? I’m curious what your baseline is. 16 GB should be enough, unless maybe you’re trying to sync/publish some really large repositories.

No, not of mine perhaps Froeman or Katello have schedule tasks in its own scheduler ?
Look at join logs, there is a task before kill process

Thanks for your reply.

Hi,

I complete my post.

I synchronize repositories all week-ends with scheduled tasks. Each Monday, pulp service is broken because mongodb is down due to this error.

`Jun  5 14:20:06 katello-04 systemd: rh-mongodb34-mongod.service: main process exited, code=killed, status=9/KILL

when I do just one repository synchronization, it’s ok

Hi @tck-lt,

Apologies for the delay. How many repositories are you syncing, and approximately how much content do they have? I have seen cases where 16 GB is not enough to publish large numbers of repositories.

Also, do you see failed tasks in the Foreman tasks monitor?

Hi,

I synchronize 36 repositories per week-end. The bigger is RHEL Server. I have increase my swap memory to 8Gb and it seems to be better now (RAM at 16Gb)

Ah ok, the RHEL server repo is a pretty big RAM consumer during a sync. The RAM usage you’re seeing doesn’t surprise me, and I’m glad the bumping up the swap helped.

1 Like