Katello 4.5/Foreman 3.3 memory leak (in gunicorn)

I now have a new colour in my chart…

If so, we have 6 - with one of those being the parent of the 5 others.

Great, so that possibly explains why it’s staggered, different processes being recycled at different times. That crazy spike remains a mystery though.

I wouldn’t think it’s the garbage collector since Python GC is mostly reference counted. There would have to be massive cycles in order for the GC to cause something like that.

@John_Beranek Did you ever attempt trimming the changelogs, whether that helps? We have other reasons to suspect that could be the issue now.

https://bugzilla.redhat.com/show_bug.cgi?id=2122344

We had not yet, but have now - we’ll see how it goes now…

$ sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' /usr/bin/pulpcore-manager rpm-trim-changelogs
System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Trimmed changelogs for 155359 packages

Possibly related to requesting lots of packages with a subset of fields still causes changelogs, filelists, requires, etc.. to be loaded, resulting in slow requests · Issue #2138 · pulp/pulpcore · GitHub

Trimming changelogs is a backdoor method of alleviating that issue (one with other benefits anyway)

@John_Beranek Let me know if you start seeing a benefit from this

Well, here’s the last 7 days - “pulp-app” peaked at 3.7GiB, “pulp-content-server” is a very steady 1.8GiB
(We disabled the unused Puppet element…well, I thought we had, but it still shows on the chart!)

Ah, we disabled Foreman/Katello support for Puppet, but forgot to disable the service - doh!

Still significant swap used, but lots of free memory and cached memory.

A peak at 3.7gb seems like a definite improvement - less than half of what it was before and not as “leaky”.

Could still possibly be improved, I think we can track that under requesting lots of packages with a subset of fields still causes changelogs, filelists, requires, etc.. to be loaded, resulting in slow requests · Issue #2138 · pulp/pulpcore · GitHub