Foreman-rake facts:cleanup batch/rolling support

Support
1

Awhile back i posted with issues running reports cleanup due to it trying to batch it all up into one transaction and timing out if it took longer than XXX seconds in the rake job.

I’m not seeing the same thing with the facts:cleanup job. I currently have the following count(s)
fact_names ‘7991166’
fact_values ‘8062049’

Running the rake task today took around 6 hours on the first phase.
I then got a Ruby Stack error. It “appears” it tried to delete all facts (by ID) in a single transaction. This transaction is longer than my console output, and i cant even grab it from the slow_query log, as it appears to be hundreds of thousands - of not millions of ID’s in a single query.

Is there a way for this rake task to also “batch up” deletions? Much like reports this may “take longer” but will also probably allow it to finish in scenarios where it will take a while to catch up or process them all?

2

Bump - would someone be able to validate (at least conceptually) that implementing the same “batch” support as was done for reports would be a viable option to remediate this?

3

https://1drv.ms/u/s!AuWxC_g_UxQokVUu0AsfYNiakUY_

Attached the console output. I had to dump it to a text file and cannot upload it here or on pastebin due to it’s size. Beware though - the single SQL query is essentially 13MB of "ID’s in the DELETE query - no wonder it’s timing out!

4

Yes, for sure. If you can confirm that the previous patch does help in production, then implementing this for facts should be very similar.

5

Drop your feedback in the PR please:

https://github.com/theforeman/foreman/pull/6153