Error reclaiming space

Problem:

I just remove the content about CentOS 8 from my Foreman 3.3 - Katello 4.5 server (to save backup space before to migrate to EL8 and getting update to date )
But trying to reclaim space from my proxy, the task fails with the following error.

Expected outcome:
Space freed

Foreman and Proxy versions:

Foreman 3.3.1 Katello 4.5.1

Foreman and Proxy plugin versions:

plugin version
foreman-tasks 6.0.2
foreman_puppet 4.0.1
foreman_remote_execution 7.2.2
katello 4.5.1

Distribution and version:
CentOS 7.9.2009

Other relevant data:

Action:

Actions::Pulp3::CapsuleContent::ReclaimSpace

Input:

{“services_checked”=>[“pulp3”],
“repository_hrefs”=>,
“smart_proxy_id”=>1,
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>“baeec99d-ee53-4b43-95c0-c3b223e3308a”,
“current_timezone”=>“Europe/Paris”,
“current_organization_id”=>1,
“current_location_id”=>2,
“current_user_id”=>4}

Output:

{}

Exception:

PulpcoreClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Wed, 11 Jan 2023 13:20:14 GMT”, “server”=>“gunicorn”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“POST, OPTIONS”, “x-frame-options”=>“DENY”, “content-length”=>“34”, “x-content-type-options”=>“nosniff”, “referrer-policy”=>“same-origin”, “correlation-id”=>“baeec99d-ee53-4b43-95c0-c3b223e3308a”, “access-control-expose-headers”=>“Correlation-ID”, “via”=>“1.1 srvtlpfk01.dit.cb”, “connection”=>“close”}
Response body: {“repo_hrefs”:[“Must not be .”]}

Hi @GC29 ,

It looks like you’re hitting Bug #34932: Space reclaiming fails on a blank Satellite - Katello - Foreman which only made it into Katello 4.6+.

The failure at least means that you have nothing to clean up on that smart proxy. If it’s blocking your other cleanups, you can go into the Dynflow console and skip the failing task.

The fix is just one line of code – if you’re feeling adventurous you could patch it in yourself. You would only need to edit the file and then restart your Foreman services.

We likely won’t do another 4.5 release since 4.6 and 4.7 are out.

If you want to patch it yourself and have questions, I can help.

Thanks for your answer!

I felt like an adventurer so I path both files:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.1/app/lib/actions/pulp3/capsule_content/reclaim_space.rb
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.1/app/lib/actions/pulp3/repository/reclaim_space.rb

and restarted foreman services.
Now the task ends with errors:

There is no downloaded content to clean.

Which is quite annoying after deleting several CentOS 8 repos…

I realize this might sound a little confusing, but space reclamation relies on your repositories actually existing. Space reclamation is mostly for getting your space back from on demand repositories.

We need to ask Pulp if there is a good way to clean Pulp artifacts for repositories that no longer exist… the answer might be that you need to resync the deleted repository (which would use the old artifacts) and then re-run disk space reclamation.

@dralley is there any way to clean up space from deleted repositories? There isn’t any automatic cleanup right?

We should add some docs and perhaps a UI warning not to delete your repositories before running cleanup.

Added Bug #35935: Add warning in UI that space reclamation won't work on deleted repositories - Katello - Foreman and Add note that space reclamation requires repositories to exist · Issue #1934 · theforeman/foreman-documentation · GitHub

Okay, so if you want to clean up the disk space for deleted repositories, you can run orphan cleanup:
foreman-rake katello:delete_orphaned_content

Orphan cleanup also runs on a weekly schedule.

It was exactly what I needed! Thanks !
The task was effectivly planned for this week-end, you saved my friday :sweat_smile:

Wouldn’t that be a good idea to trigger this task after repository deletion? (at the end of the day maybe)

1 Like

Awesome to hear it helped!

Orphan cleanup can eat up some compute and memory resources, so we’ve opted to run it by default on the weekend when there might be less traffic to Foreman. I’ll add that (for now) it’s on a Cron schedule, so you could change that yourself if you wanted something more frequent.

I suppose that default is as it is because we usually don’t anticipate users having tight disk space requirements. If we did, then it might make more sense to have the cleanup be triggered after repo deletion.

1 Like