Clean up orphaned facts

Support
1

I’m trying to clean up orphaned facts from a host that’s used as a CI-runner. This host gets new interface facts for each docker instance that it starts, so now we’re up to over 1500 facts simply for interfaces! I’ve tried to clean up orphaned jobs with " foreman-rake facts:clean" but I believe the I’m going over the limit of facts that can be cleaned:

/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'

Caused by:
PG::UnableToSend: number of parameters must be between 0 and 65535
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'

Any ideas on how to get to the bottom of this? I simply would like to delete all the facts from these hosts and start over.

Foreman and Proxy versions:
Foreman 1.20.2
Proxy 1.20.2

Foreman and Proxy plugin versions:
Puppet Smart Proxy 1.20.2

2

Known limitation. Upgrade to 1.23 and then do this: foreman-rake facts:clean batch_size=500 (you can play around with the batch size to find the best value for you).

Or you can backport Feature #25227: Foreman-rake facts:cleanup batch/rolling support - Foreman

3

Thanks. Tried this after looking through the code on 1.20. Now I know why it didn’t work!

4

Unfortunately, I’m running into problems with this. Any ideas?

# foreman-rake facts:clean batch_size=10 --trace
** Invoke facts:clean (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute facts:clean
Starting orphaned facts clean up
rake aborted!
NoMethodError: undefined method `map' for nil:NilClass
/usr/share/foreman/app/services/fact_cleaner.rb:54:in `block in delete_excluded_facts'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:222:in `loop'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:222:in `in_batches'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/usr/share/foreman/app/services/fact_cleaner.rb:53:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:10:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:15:in `block (2 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/usr/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/usr/bin/rake:27:in `<main>'
Tasks: TOP => facts:clean
5

It’s a bug: https://github.com/theforeman/foreman/pull/7185