Clean up /var/lib/pulp/media/upload after failed upload

Problem:
Hi,

Earlier this week an upload of CentOS 7 iso failed with the message “Too many open files”
So I configured the # open files for the user pulp with 1024.
However, this was still not enough, so I increased it to 8192 in orde to make it work (afterwards it have set it to soft 2048 and hard 4096 which I hope should be more than enough).
With this new setting I was able to successful upload the CentOS 7 iso.

Then I started the upload of CentOS 8 iso.
This one failed several times due to insufficient disk space.
First I thought it was caused by the small /var I have configured in LVM.
I have separated LVM volumes for /var/lib/pgsql, /var/log, /var/lib/qpidd and /var/lib/pulp.
So, I increased the size for /var 2 times (5 -> 15GB, and afterwards to 20GB), but it still complained for insufficient disk space.
Finally I found out the problem was caused by /tmp, which was part of the root-fs that was configured in LVM with a size of 5GB.
Well, I created a new /tmp LVM volume with a size of 15GB.
And yeah, with this new /tmp the upload of the CentOS 8 iso went well.

But meanwhile I could also see that for each upload command I have executed with hammer, /var/lib/pulp grow in use.
For each upload new files were created in /var/lib/media/upload.
Those from a failed upload were not cleaned up.

So, I would like to know if there is a simple way/command to clean up those files from the failed upload-commands?

This all happend on a clean foreman 2.4 / Katello 4.0 install on a brand new system running CentOS 8.

Thanks in advance for any feedback.

Expected outcome:

Clean up of uploaded files after an upload- failure

Foreman and Proxy versions:

Foreman 2.4
Katello 4.0

Foreman and Proxy plugin versions:

Distribution and version:

CentOS Linux release 8.3.2011

Other relevant data:

This appears to be related to a bug in pulp3 - Issue #7316: Files are not being deleted from storage when calling the method delete() - Pulp

1 Like