Problem:
Hey,
we noticed a problem when querying the Pulp PyPI simple index (`/simple//`) on a repository containing around 100,000 Python packages.
pip install takes an unusually long time and regularly results in timeouts.
In addition, we’ve noticed that the server is using an enormous amount of RAM.
The root cause is in pulp_python/app/pypi/views.py, where if content:is used on an unfiltered Django QuerySet:
if content:
local_packages = content.filter(...)
This results in 100,000 Python packages being loaded into PostgreSQL before the .filter() is even reached.
For us, this sometimes led to host outages when multiple Python queries were executed simultaneously.
Upstream pulp_python resolved this in version 3.28 (commit 6adc131716156cab30318a3463f45b6655324215) by changing the check to if content is not None:. However, python-pulp-python.spec in foreman-packaging is currently still at 3.27.6.
We manually tested the extension of the condition with “is not None”
This resolved all issues and sped up the download of the packages.
Expected outcome:
Could python-pulp-python in python-pulp-python.spec be bumped from 3.27.6 to 3.28 (e.g., 3.28.0 / 3.28.2) for our Katello release branch (or the if content is not None: patch be backported to 3.27.6 if a minor bump conflicts with pulpcore dependencies)?
Foreman and Proxy versions:
The Versions we use are:
Foreman version: foreman-proxy-3.19.1-1.el9.noarch
Katello version: katello-4.21.1.1-1.el9.noarch
However, the bug will likely also occur in versions 5.0 and 5.1, since the pulp-python version 3.27.6 comes from pulpcore/3.105;
a higher version is only included in pulpcore/nightly.
Distribution and version:
Rocky 9.8 & Rhel 9
Other relevant data:
(New to the forum… I hope this topic fits here
)