Request to bump python-pulp-python from 3.27.6 to 3.28 in python-pulp-python.spec (PyPI simple index timeout / RAM spike)

Problem:

Hey,

we noticed a problem when querying the Pulp PyPI simple index (`/simple//`) on a repository containing around 100,000 Python packages.

pip install takes an unusually long time and regularly results in timeouts.
In addition, we’ve noticed that the server is using an enormous amount of RAM.

The root cause is in pulp_python/app/pypi/views.py, where if content:is used on an unfiltered Django QuerySet:

if content: 
    local_packages = content.filter(...)

This results in 100,000 Python packages being loaded into PostgreSQL before the .filter() is even reached.
For us, this sometimes led to host outages when multiple Python queries were executed simultaneously.

Upstream pulp_python resolved this in version 3.28 (commit 6adc131716156cab30318a3463f45b6655324215) by changing the check to if content is not None:. However, python-pulp-python.spec in foreman-packaging is currently still at 3.27.6.

We manually tested the extension of the condition with “is not None”
This resolved all issues and sped up the download of the packages.

Expected outcome:

Could python-pulp-python in python-pulp-python.spec be bumped from 3.27.6 to 3.28 (e.g., 3.28.0 / 3.28.2) for our Katello release branch (or the if content is not None: patch be backported to 3.27.6 if a minor bump conflicts with pulpcore dependencies)?

Foreman and Proxy versions:

The Versions we use are:

Foreman version: foreman-proxy-3.19.1-1.el9.noarch

Katello version: katello-4.21.1.1-1.el9.noarch

However, the bug will likely also occur in versions 5.0 and 5.1, since the pulp-python version 3.27.6 comes from pulpcore/3.105;
a higher version is only included in pulpcore/nightly.

Distribution and version:

Rocky 9.8 & Rhel 9

Other relevant data:

(New to the forum… I hope this topic fits here :sweat_smile: )

Hi @Muvcuv ,

I think we’ll have more success asking for the backport of that commit into pulp-python 3.27. We have to be careful when bumping y-versions of Pulp plugins since, sometimes, things change enough that we need to re-certify that the plugin is compatible with Katello.

I am asking the Pulp team about getting this backported on the commit and in the pulp-dev matrix channel.

If it’s backported, it should be quite straight-forward for us to package it and ship it out in the 3.105 branch.

cc @Odilhao

Getting things rolling: Update pulp-python to 3.27.9 by ianballou · Pull Request #3412 · theforeman/pulpcore-packaging · GitHub