as Foreman grows we have more and more fact parsers. Today it’s at least:
- Puppet facter
- Discovery facter (based on Puppet)
- Red Hat Subscription Manager facts
- Ansible facts
- Salt facts
- custom plugins
The idea is to gather those facts and show them in Foreman inventory, that is one of the main features of Foreman. Now, during fact upload we parse some facts and detect several key things:
- puppet environment
- hardware model
- NICs (regular, virtual, bond, VLAN, bridge, IPMI)
The problem is that historically the parsing code was designed for Puppet and then we started creating more common parser and breaking things into various implementations: Puppet, RHSM, Ansible, Salt, Discovery etc. The common parser is not very common, it is still designed for Puppet, so other parsers needs to build to of that and do various hacks/workarounds in order to deliver detection of the above.
For example Puppet facter converts virtual (VLAN) interface from
bond0_123 format and this is not gonna change upstream (patch was denied, breaking change they can’t do anymore). But underscore is also used for bridge (virbr-1_nic). This leads to various hacks and regexp matching in order to get what we need.
Another problem is when users are using more fact reporting clients simultaneously. Different clients reports OS or NICs differently, so when Puppet checks in a host is associated with “Redhat 7.5” but when RHSM checks in its “Red Hat Enterprise Linux 7.5”. Or Puppet and Ansible - they can “fight” overriding detected fields over and over again which slows down the system.
Speaking about performance, fact processing is currently slow. It’s one of the major performance points for large-scale deployments. If fact client is under our control, we can easily implement fact “caching” and only send those which changed. This will bring processing times down by order of magnitude.
Key areas to improve
- Simplify fact parsing code
- Simplify NIC detection code
- Design fact names and format in Foreman-friendly way
- Get facter client under our control (regressions or changes in facter, ansible, salt, rhsm)
- Faster fact processing
- One parsing code to maintain in the future
I am thinking on creating our own “facter” client that would deliver minimum set of facts reported with “foreman_” prefix which would be complementary to what we have today. We can start as easy as writing a shell script which will execute from cron and upload facts to HTTPS/JSON endpoint hourly.
We do not need all puppet facts at all, we are mostly interested in the facts that allow detection of the above: NIC, VLANs, bonds, domain, subnet, architecture and hardware model. We would not report Puppet environment, that would remain job for puppet parser.
We would need to write new fact parser implementation to detect and parse “foreman_” facts. This would be opt-in via setting, so users can test this and provide feedback. We would provide long transition phase, new facts would appear alonside with Ansible/Puppet/Salt facts. Once we are sure this is a good way to go, we can start removing other parsers. We would still support importing of Puppet, RHSM, Ansible, Salt and other facts, but they would not be parsed anymore - just stored for the inventory purposes and config management.
Discovery that’s another story, we have dozen of custom facts and hacks in common parsing code in order to deliver required features. Once foreman facter is implemented, discovery could take advantage of that. We could have a common fact reporting client for both discovered nodes and managed/unmanaged nodes - LLDP network discovery currently present in FDI could be used in core.
We are already talking about moving puppet into plugin long term - such a refactoring would be much easier if we had our parsing code not tightly coupled with puppet.
This is core part of Foreman so evolution is the key, therefore I propose the following process:
- Create new parser called ForemanFactParser
- Provide new Global setting “Use Foreman parser for OS, Subnet and NIC detection” (probably we can break this into individual settings so users can opt-in one after another)
- Write a shell script that will provide the facts to foreman
- Provide provisioning template which will install and enable the shell script via cron
- Common fact reporting for discovered, managed and unmanaged hosts
Going forward, we can turn the shell script into something more powerful or rewrite this into compiled language so there is no additional dependency on registered hosts (let’s avoid Ruby or Python). Things like throttling for large-scale deployments can be easily implemented if we build this into the API (“slow down I am busy, next checkin is in 4 hours instead of one hour”). But let’s start small.