Foreman 1.24 will drop too many facts

All,

Foreman no longer imports all facts, structured facts (e.g. networking::interface) and selected root facts:

  • blockdevice_
  • ipaddress6?_
  • macaddress_
  • mtu_
  • speed_
  • auto_negotiation_
  • duplex_
  • link_
  • wol_

are only imported up to 100 elements. If a host reports more, they will be dropped and fact named foreman::dropped_subtree_facts will contain approximate number of facts dropped (e.g. 2 thousands). The maximum amount of facts is configurable via Administer - Settings - Provisioning. This change was implemented for large deployments with many nodes and/or hypervisors or container hosts with many virtual network cards, bridges, mountpoints or volumes.

We believe that limit of 100 is a sane default, Foreman users usually don’t work with hundreds of NICs or mountpoints. If needed, this limit can be increased however severe performance penalties across whole application can be expected as the database and indices grow.

The relevant change is:

I filed an upgrade note as well. There is nothing to do during upgrade, old facts will be dropped on the fly as managed nodes will be checking in and filter will take effect. Initial checkins might be little bit slower than usual as there will be some records to delete.

I like that you write the count of dropped facts to a “fact” :slight_smile:

This should make tuning/customization workable too. I can dump the contents of that fact across my environment and determine a percentile that works for overall inclusions vs performance. Something like 95th percentile or something probably.

Should there be a whitelist/override inclusion list and/ or should the attribute controlling the import recursion number be overrideable through the ENC? I personally see no need for it, but I could imagine an environment heavy on docker having many hosts with many network interfaces being a “thing” and to be able to allow only that fact entirely, or a larger recursion limit on a specific hostgroup to be useful features, particularly in a larger environment.

There is not such thing at the moment. Honestly I tried to design this as simple as possible because we already have NIC and fact name exclude filter. This would need fact name include filter, it’s a mess. Let’s attack this if there is such a request.