RFC: Foreman facter

Hello,

as Foreman grows we have more and more fact parsers. Today it’s at least:

  • Puppet facter
  • Discovery facter (based on Puppet)
  • Red Hat Subscription Manager facts
  • Ansible facts
  • Salt facts
  • custom plugins

The idea is to gather those facts and show them in Foreman inventory, that is one of the main features of Foreman. Now, during fact upload we parse some facts and detect several key things:

  • operatingsystem
  • puppet environment
  • architecture
  • hardware model
  • domain
  • subnet
  • NICs (regular, virtual, bond, VLAN, bridge, IPMI)

The problem is that historically the parsing code was designed for Puppet and then we started creating more common parser and breaking things into various implementations: Puppet, RHSM, Ansible, Salt, Discovery etc. The common parser is not very common, it is still designed for Puppet, so other parsers needs to build to of that and do various hacks/workarounds in order to deliver detection of the above.

For example Puppet facter converts virtual (VLAN) interface from bond0.123 to bond0_123 format and this is not gonna change upstream (patch was denied, breaking change they can’t do anymore). But underscore is also used for bridge (virbr-1_nic). This leads to various hacks and regexp matching in order to get what we need.

Another problem is when users are using more fact reporting clients simultaneously. Different clients reports OS or NICs differently, so when Puppet checks in a host is associated with “Redhat 7.5” but when RHSM checks in its “Red Hat Enterprise Linux 7.5”. Or Puppet and Ansible - they can “fight” overriding detected fields over and over again which slows down the system.

Speaking about performance, fact processing is currently slow. It’s one of the major performance points for large-scale deployments. If fact client is under our control, we can easily implement fact “caching” and only send those which changed. This will bring processing times down by order of magnitude.

Key areas to improve

  • Simplify fact parsing code
  • Simplify NIC detection code
  • Design fact names and format in Foreman-friendly way
  • Get facter client under our control (regressions or changes in facter, ansible, salt, rhsm)
  • Faster fact processing
  • One parsing code to maintain in the future

Proposal

I am thinking on creating our own “facter” client that would deliver minimum set of facts reported with “foreman_” prefix which would be complementary to what we have today. We can start as easy as writing a shell script which will execute from cron and upload facts to HTTPS/JSON endpoint hourly.

We do not need all puppet facts at all, we are mostly interested in the facts that allow detection of the above: NIC, VLANs, bonds, domain, subnet, architecture and hardware model. We would not report Puppet environment, that would remain job for puppet parser.

We would need to write new fact parser implementation to detect and parse “foreman_” facts. This would be opt-in via setting, so users can test this and provide feedback. We would provide long transition phase, new facts would appear alonside with Ansible/Puppet/Salt facts. Once we are sure this is a good way to go, we can start removing other parsers. We would still support importing of Puppet, RHSM, Ansible, Salt and other facts, but they would not be parsed anymore - just stored for the inventory purposes and config management.

Discovery that’s another story, we have dozen of custom facts and hacks in common parsing code in order to deliver required features. Once foreman facter is implemented, discovery could take advantage of that. We could have a common fact reporting client for both discovered nodes and managed/unmanaged nodes - LLDP network discovery currently present in FDI could be used in core.

We are already talking about moving puppet into plugin long term - such a refactoring would be much easier if we had our parsing code not tightly coupled with puppet.

Execution

This is core part of Foreman so evolution is the key, therefore I propose the following process:

  • Create new parser called ForemanFactParser
  • Provide new Global setting “Use Foreman parser for OS, Subnet and NIC detection” (probably we can break this into individual settings so users can opt-in one after another)
  • Write a shell script that will provide the facts to foreman
  • Provide provisioning template which will install and enable the shell script via cron
  • Common fact reporting for discovered, managed and unmanaged hosts

Going forward, we can turn the shell script into something more powerful or rewrite this into compiled language so there is no additional dependency on registered hosts (let’s avoid Ruby or Python). Things like throttling for large-scale deployments can be easily implemented if we build this into the API (“slow down I am busy, next checkin is in 4 hours instead of one hour”). But let’s start small.

1 Like

Note that Ansible uses facter (which is a standalone project originating from Puppet) if it’s available.

Facter 3 has the structured fact networking which is very much reworked. Has anyone with bonding actually looked at the result of it?

:-1: from me. I don’t want to require anything to be deployed on client systems. In the current Puppet world the client doesn’t talk to Foreman - just the Puppet master.

Has your approach also considered that the Puppet ENC output is determined based on facts? AFAIK the current workflow is that node.rb sends the facts to Foreman, that updates the facts and calculates the correct parameters for the node. Your workflow would break this. This is also the reason why you can’t slow down the update: you’d be telling the client ‘do your puppet run in a few hours’.

1 Like

How Facter 3 would help us with getting rid of Puppet as required component to do detection of host fields in Foreman? This RFC is actually all about giving our users ability to use Foreman without Puppet, Facter or PuppetMaster.

That’s Puppet world.

There are two separate things:

  • Puppet facts
  • Parsing of puppet facts

My proposal does not want to change the former, puppet facts are still uploaded, ENC works. It’s just the puppet NIC/OS/subnet/etc parsing code which could be turned off and provided by different client. Completely optional.

I absolutely do not want to ruin experience for existing Puppet users, there no doubt about this. I just don’t see at this point how I am breaking it. Help me to understand what’s on your mind and thanks for the feedback.

How does Ansible do fact collection today? It seems like YAFC (Yet Another Fact Collector) is not a great use of time.

It has its own module call “setup” and provide facts in completely different format. Our fact parser needs to deal with that. Edit: If facter is present, it also calls it and reports them and Ansible plugin take use of some of them.

We already have update_subnets_from_facts and update_environment_from_facts settings. In Fixes #23683 - Fix importing os fact by mdellweg · Pull Request #5610 · theforeman/foreman · GitHub I suggested to generalize this to an array update_from_facts.

That would change it more into a pipeline:

fact reporterfact updaterhost updater

Note that we already have the concept of fact sources in Foreman which means the reporter and updater in the pipeline are already covered. I’d say the updater part just needs to be brought in line with the rest.

Your solution adds another reporter and reminds me of:

2 Likes

I am open to suggestions how we turn the fact parsing spaghetti code into something more useful.

No need of making fun of this, I am trying to improve. It’s different angle, something you would not consider at all perhaps. I am not seeking any kind of standard here, just something that would fit our use case and work without any Puppet/Ansible/other clients.

I like the idea behind of the proposal as I had multiple issues when having as Ansible environment. Typically facter is not installed then, not every playbook has to run setup/fact gathering or can only do a subset of facts.

On the other side requiring another tool to be installed before registration would perhaps make things more complicated as it must be available as package for all possible operating system (including Windows) or if not packaged for prepared for all config management solutions.

I’m not sure yet, what I think about this. :slight_smile:
But I am wondering how we can safely identify a host. How does Foreman now what host is reporting facts? In the puppet world, we have certificates provisioned on the hosts. Do we need our own CA in Foreman?

btw: We have a fact sending client written in golang (called blossom). It can get a certificate from puppet ca, compile facts about the system, and upload them to foreman (via a custom fact parser :-P). I can check if we can open source that if you’d be interested. It runs very smoothly in a docker container but can be deployed standalone.

2 Likes

Maybe we should think about that CA as an independent service provided by foreman. That way it could be used by the facts aggregator as well as any other plugin. For example a “package upload profile” which abuses the subscription-manager CA atm.
And in a perfect world, puppet, ansible, you name it could be configured to use that CA.

I’m not trying to make fun of it, but to me it’s unclear exactly which problem you’re trying to solve. To quote your Key areas to improve:

I think that when you suggest a new fact producer the XKCD applies for every single point.

If you’re trying to solve the problem that you want to produce facts for a system without puppet/ansible/subscription manager/salt then a new fact producer can be a valid solution but that’s not the problem you describe.

Your suggestion implies that you want to move the fact parsing to another system and that can be valid (a proxy endpoint sounds reasonable) but essentially you’re moving the problem to another project. I’m tempted to quote “All problems in computer science can be solved by another level of indirection” but here I do see benefits in indirection because the proxy solves the abstraction problem for many of our use cases already (see DNS/DHCP) to keep Foreman Core lighter. We could also use the already present authentication from proxy to foreman in some cases.

In the Puppet world we do. node.rb uses the Proxy’s client certificate to authenticate and inside the actual call it identifies which node is there. The Foreman then trusts the Proxy to identify the client system.

As @ekohl said before, Facter is a project developed by Puppetlabs, but it has zero connection to puppet. https://github.com/puppetlabs/facter

It has it’s own module (https://docs.ansible.com/ansible/latest/modules/setup_module.html) but if Facter or Ohai are installed it also takes advantage of that and gathers these facts

Facter does work without any of these!

I’m sorry but I really fail to see why points are a pain point right now, or why another fact producer would fix it. Except for ‘getting facter client under our control’ which IMO is not enough benefit for the cost. Particularly the last one is not true, it’s important for us to parse Facter/Ohai.

In general, I would say that Foreman should try to meet users where they are. Meaning we should adapt to them, not make them use another tool (which we have to develop & maintain).

Facter has a stellar track record, it’s very fast, and it has a pluggable architecture, many people already use custom facts and have invested time into writing them to work with Facter. No wonder, it’s a mature tool. I would not want us to spend time doing basically what this tool does already.

If you want new facts or improve the format of the current ones, I would push for deploying custom facts (https://github.com/puppetlabs/facter/blob/master/Extensibility.md#custom-facts-compatibility) or improving the existing ones.

@lzap I appreciate you’ve had troubles with things like ‘bond0.123 to bond0_123’. In my opinion, it’s far easier and better for our users to hack and regexp match whatever is needed, or even provide a custom fact for bonded interfaces (last resort), than writing our own Foreman Facter from scratch - even maintaining a fork would be a bad idea if you ask me.

1 Like

Thanks for the feedback everyone. I stepped too far away from my comfort zone of provisioning and I was a bit frustrated reading our parsing code. It needs an overhaul for sure some day.

2 Likes

Seems like a consensus has been reached before I could reply - but I’ll add a big :+1: for making the Importers (facts & reports) more “pluggable” and/or robust, at least. I agree that we don’t need another fact collector, but making the process of creating/maintaining new ones smoother can only be a good thing?

(this post brought to you by yet-more-bugs-in-the-salt-importer-dot-com :stuck_out_tongue:)

1 Like

I created “rejected” tag and tagged this flag.

Is my understanding of current puppet parsing code in Foreman correct: we use “old” facts only while puppet3+ can also send “new” format (networking::slight_smile: which can deliver more accurate info?

Then as part of the refactoring, we should create two separate parsers perhaps, puppet2 and puppet3 and detect which one to use? Would this be a good idea at all?

1 Like

Perhaps this just needs to be more user-configurable - EG, use a config file or DB config to allow users to select the data source for a field/fact - EG, facter, discovery, salt grains, RHSM, etc. This could also provide a field to allow the conversion of facts - eg, use Ruby’s stream editing ability to convert things like “Red Hat Enterprise Linux 7.5” to “Redhat 7.5” or vice-versa based on the users desires. This allows Forman to have a default setting, but to also be configurable and extendable in terms of fact management to make it more flexible.

It might also be useful to provide the ability to add or update a random fact via the api (via the v2 or v3 api) - this could even allow for user-defined facts in Forman.

1 Like