Cannot clean old repository packages

Support
,
1

Problem:
On my test instance I downloaded two Ubuntu repos with “Immedate” policy. After tests I want to reclaim my space and deleted all related CVs, repositories and products with success. Still I have 88% disk usage when I should have almost nothing.
I canceled sync of one of repos because it was hitting 98%. After hitting cancel button it immediately dropped to 88% and I cannot get rid of it.

Expected outcome:
/var/lib/pulp storage freed up

Foreman and Proxy versions:
Single instance, foreman version 3.14
Foreman and Proxy plugin versions:

*foreman-release-3.14.0-1.el9.noarch*
*foreman-installer-3.14.0-1.el9.noarch*
*rubygem-foreman_maintain-1.10.3-1.el9.noarch*
*foreman-installer-katello-3.14.0-1.el9.noarch*
*foreman-selinux-3.14.0-1.el9.noarch*
*foreman-3.14.0-1.el9.noarch*
*foreman-postgresql-3.14.0-1.el9.noarch*
*foreman-service-3.14.0-1.el9.noarch*
*foreman-dynflow-sidekiq-3.14.0-1.el9.noarch*
*foreman-redis-3.14.0-1.el9.noarch*
*rubygem-foreman-tasks-10.0.2-1.fm3_14.el9.noarch*
*rubygem-foreman_remote_execution-15.0.2-1.fm3_14.el9.noarch*
*foreman-proxy-3.14.0-1.el9.noarch*
*rubygem-hammer_cli_foreman-3.14.0-1.el9.noarch*
*foreman-cli-3.14.0-1.el9.noarch*
*rubygem-hammer_cli_foreman_tasks-0.0.22-1.fm3_14.el9.noarch*
*rubygem-hammer_cli_foreman_remote_execution-0.3.2-1.fm3_14.el9.noarch*
*ansible-collection-theforeman-foreman-5.1.0-2.el9.noarch*
*rubygem-foreman_ansible-15.0.7-1.fm3_14.el9.noarch*

Distribution and version:
Foreman hosts installed on Oracle Linux 9.5
Other relevant data:
Tried foreman-rake katello:delete_orphaned_content and also with RAILS_ENV=production, nothing helped.

2

It is possible you are running into the orphan protection time. By default orphans that are not yet older than the orphan protection time, are not deleted by orphan cleanup. This is to prevent orphan cleanup from deleting packages before pulp has had a chance to add them to a repo. I forget what the default orphan protection time is. There is also an option to run orphan cleanup with a protection time of zero (but I don’t remember the exact command). Alternatively, wait for the orphan protection time to run out and then re-run orphan cleanup.

The only other possibility is that the Ubuntu packages are still part of a repo, but this is not possible if you have successfully deleted your Ubuntu repositories as stated.

3

You must be patient. pulp has it’s own cleanup schedule. I think it keeps orphaned artifacts for at least 24 hours.

4

Okay, I will wait till tomorrow and post an update, thanks!

5

It’s in the pulp api: REST API - Pulp Project

ORPHAN_PROTECTION_TIME is 24 hours by default.

You can see when it was executed with

curl --cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://foreman8.example.com/pulp/api/v3/tasks/?name=pulpcore.app.tasks.orphan.orphan_cleanup' | jq

You can execute the orphan cleanup directly like this:

curl -X POST -H 'Content-Type: application/json' --cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://foreman8.example.com/pulp/api/v3/orphans/cleanup/' -d '{"orphan_protection_time":720}'

USE AT OWN RISK!

It will return the URI for the task created. Set the time as you need. It’s in minutes, i.e. 720 minutes aka 12 hours.

Be careful with too low numbers on production systems. As the apidoc says:

The time in minutes for how long Pulp will hold orphan Content and Artifacts before they become candidates for deletion by this orphan cleanup task. This should ideally be longer than your longest running task otherwise any content created during that task could be cleaned up before the task finishes. If not specified, a default value is taken from the setting ORPHAN_PROTECTION_TIME.

If you pick a very short time it may remove the content which is currently synced in…

6

You were right, I ran cleaning orphans again today and it did clean repos, thank you!

7

If you don’t run it manually, it’s usually called from /etc/cron.d/katello running every Sunday night.

8

Yes, I got this command initially from cron, thanks for clarification