Problem: Remove orphans tasks are in paused state for last 2 days
Expected outcome: Remove orphans tasks are getting completed successfully
Foreman and Proxy versions: Foreman 3.13 Katello 4.15
Foreman and Proxy plugin versions: Foreman 3.13 Katello 4.15
Distribution and version: Rocky 9
Other relevant data:
Remove orpans tasks are in paused state from last 2 days. When we try to resume it also, its getting paused again with the below error:
Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Tue, 08 Apr 2025 11:59:10 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, DELETE, HEAD, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"174", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "cross-origin-opener-policy"=>"same-origin", "correlation-id"=>"33ad482c95bf463a86a95ff0cdf426b8", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 example.com", "connection"=>"close"}
Response body: ["The repository version cannot be deleted because it (or its publications) are currently being used to distribute content. Please update the necessary distributions first."]
Please let us know is there any other logs are required for further troubleshooting.
This typically indicates that there is a Katello repository somewhere with inconsistent version_href and publication_href stored in the DB. This usually happens because some sync was aborted after the version_href was updated, but before the publication_href was updated. (We should make it so these are only ever updated together once both are available).
The only way to get orphan cleanup to work again is to find and fix the affected Katello repo.
If it is not yet in any content view versions, then it should be fixed with the next successful sync.
The hard part is figuring out what repo is the affected repo.
Do you see any failed sync tasks for any of your repos? If so try re-syncing those repos and see if that helps.
We’ve made some updates recently that should help with this. There are a number of ways the issue can crop up. The gist of why it is happening is that Pulp somewhat recently added new restrictions on deleted repository.
This second change fixes orphan cleanup in a few ways. On smart proxies and the main server, distributions that Katello lost track of are now fully removed. Orphan repository version cleanup now comes later, so orphaned repositories and distributions can get cleaned first, which should avoid some cases of the error here.
We’ve also added logic so orphan cleanup is not paused by this error. In some cases this issue is unavoidable, for example if a sync or a CV publish fails midway and leaves hanging data. In this case, we collect information about the broken repositories / content views and give some tips for how to fix the issue.
The pull request for the second issue in particular has some design discussion that might be of interest.
The first thing to try is finding the related repository or CV (if there is one) and resync it or republish the metadata. If no CV or repo is involved and the content is truly orphaned, then the distribution in Pulp that is distributing the orphan repository version needs to be deleted first. That’s what the links I sent help with.
Next, search /var/log/messages for the Pulp repository version (version_href) that Katello tried to delete that triggered the error.
Then the CLI can be used to query the Capsule:
# To inspect any Pulp entity
pulp --profile proxy show --href <href from logs>
# Find the distribution related to the repository version by first locating the related publication (if yum repo) and then by finding the related distribution. It's possible this info may be in the error in /var/log/messages as well
pulp --profile proxy rpm publication list --repository-version <href that failed orphan cleanup>
# It seems Pulp does not support searching for distributions by publication, so that will need to be manually found.
# To list yum/rpm distributions with limited information:
pulp --profile proxy rpm distribution list --field base_path --field pulp_href --field repository_version --field publication
# Then, delete the distribution (and potentially publication):
pulp --force --profile proxy rpm publication destroy --href <publication href>
pulp --force --profile proxy rpm distribution destroy --href <distribution href>
Once this is done, the next orphan cleanup run should delete the previously problematic repository version.
I’d recommend taking some time to get familiar with the Pulp entities, it can be helpful anyway as a Katello admin.
