Following up on the announcement by @Marek_Hulan, responses to surveys and discussions with various people, this RFC outlines the technical aspects of making Puppet an optional part of Foreman and the proposed changes.
This document tries to capture the outcomes of all the discussions we have had so far on the matter, but we might have missed some things, or made wrong assumptions and choices on the planned approach - so input from more people is greatly appreciated!
I would like to thank everyone who has been involved in the discussion so far for their valuable feedback.
The two guiding principles while creating this proposal were:
- Users of Puppet should not see a deterioration in their workflows - ideally, some workflows would even see an improvement.
- Users who don’t use Puppet will have a simpler, easier to use experience with Foreman.
With Puppet 5 reaching EOL in November 2020, we are also planning on taking advantage of this change to clean up code used for supporting older versions of Puppet. We have already started on this effort with the removal of Smart Variables in 2.0, but there is much more that can still be improved.
Since this is a very large change, it will take several releases to achieve, and the outcome of this work will result in Foreman 3.0. All significant breaking changes will only be done as part of the major release change, so users will have enough time to prepare and plan accordingly.
We have identified several main workflows or integration points with Puppet. This is how we plan on addressing each one:
Facts and Reports
This is the most common use case for all configuration management solutions, and a core use case of Foreman. Being able to import and parse reports and facts from multiple sources should be a part of Foreman core.
Puppet fact and report importing and parsing will remain in Foreman core.
Some other common sources will be migrated to core: initially, Ansible and Subscription Manager seem to be the most common ones according to preliminary data from the community survey. Others may be added as well if there is enough demand.
Fact parsing of facts from legacy Facter versions (2 or older) will be dropped, and the interface between fact importers and parsers will be better defined. Perhaps some of the tasks could be moved to background processing using dynflow.
The plugin interface will continue to be maintained, to allow for adding additional sources from plugins.
All fact and report import actions should be done via a smart proxy. The proxy will handle the authentication to Foreman as well as receiving the input from the external services providing them. We will also look into possibility of receiving facts and reports directly from the Puppet server instead of using the existing scripts which require additional setup for external Puppet servers.
There is some ongoing discussion regarding which facts should be parsed and how. I expect the outcomes of that discussion to also affect the changes done in this regard.
ENC (External Node Classifier) and related workflows
This refers to a set of association between hosts and Puppet entities - namely, Puppet environments, Puppet classes and Puppet class parameters. While these are commonly used, they are used less often than facts and reports, and add a lot of complexity for users who do not use Puppet or use Puppet only as an information source for Foreman while managing its ENC functionality elsewhere (e.g. using hiera).
This functionality will be extracted to a new plugin, which will be installable without losing existing data where it exists.
As part of the extraction, we will try to incorporate some improvements as well - for example, importing parameter types for smart class parameters when possible. This will also be a good chance to consider how we can improve the workflows around this area, as well as potentially adding integrations with other tools in the Puppet ecosystem where appropriate.
Puppet module content management with Katello
This workflow is not very common, as the Puppet ecosystem has other common tools for managing modules (such as r10k).
Since Pulp 3 does not support the Puppet content type, support for the Puppet module content type has been officially deprecated in Katello 3.15 and will be removed in a future release (likely, Katello 4.0).
Puppet CA certificate autosigning for new hosts
There are currently two approaches for handling autosigning of newly provisioned hosts by the Puppet CA.
The hostname whitelisting approach, which is less secure and requires modifying Puppet configuration files directly, will no longer be supported.
The token-based whitelisting approach will continue to be supported as part of Foreman core, but the cert-based implementation that is used on Puppet 5 or older will also be removed in favor of the API-based implementation that is in use for Puppet 6.
There are currently multiple methods of executing one-time Puppet agent runs, which are difficult to set up properly, and are not very widely used. This has already been discussed over a year ago.
The approach outlined in that discussion is the one we plan to pursue - remove the various Puppet run providers in favor of the existing Remote Execution template as the recommended method of triggering a Puppet agent run on a host.
Installer setting up Puppet Server and CA
While it is currently possible to optionally disable automatic setup of Puppet Server and CA by the installer, in a Foreman installation without Katello we depend on the Puppet CA to provide and manage some certificates used by Foreman and the smart proxies.
In the shorter term, there is ongoing discussion regarding how to better handle certificates in general, including dropping this dependency. Once that work is done, Foreman will no longer require a Puppet CA be set up for installation, and making a Puppet-less installation scenario will be easier.
In the longer term, since Foreman is usually installed in a brown-field environment, we may consider no longer managing external services (such as Puppet Server and CA) using the Foreman Installer and rather only providing the installer with credentials required for communicating with such. That, however, is a separate discussion that will not be addressed as part of the current effort.
There are multiple widgets on the dashboard related to configuration management status and reports. When using more than one solution, some widgets are multiplied while providing little value.
The status related widgets will be modified to provide information regarding all host statuses, not just configuration status.
The configuration report related widgets will be modified to display consolidated information regarding all sources, and will only be displayed if there are reports viewable by the current user - leading to a cleaner dashboard for users who don’t import configuration reports to Foreman or lack permissions to see such.
Host detail page
The host view today consists mostly of two large graphs showing Puppet run statistics. These are not very useful, take a large amount of “real estate” on the screen, rely on an old graphing library we want to remove and are blank for hosts that don’t use Puppet.
This page will be redesigned in a modular way, that will display more complete information (such as details currently only viewable in the edit page) and will allow easy extensibility for plugins that wish to provide additional information. As a side effect, the Content Host view from Katello might be incorporated into this page, bringing us one step closer to the full unification of Hosts and Content Hosts.
There are currently multiple settings under the “Puppet” category that are not Puppet-specific (such as fact-related ones), and there are settings in other categories that refer to Puppet even though they may (or may not) affect other configuration management options.
The settings will be re-arranged and better described, and in some cases settings that aren’t useful may be removed.
Statistics and Trends
Most of these currently rely on Puppet related information - either fact data or ENC configuration. Additionally, they are not heavily used, as there are much better solutions for monitoring real-time statistics and trends of hosts out there.
These pages will be extracted to a new plugin, where it will be easier to improve them, and perhaps integrate them with other monitoring solutions for a better overview. This will also provide us with a good opportunity of understanding what is required to extract core functionality into a plugin, before attempting the much more complicated Puppet ENC functionality extraction.
Please share any feedback you have regarding the workflows and direction outlined above. Did we miss something? Is there some opportunity for improvements? Are we making a huge mistake? Do you like where this is going?
The whole effort will be tracked on Redmine, where it will be broken down into smaller chunks and tasks. We will gradually start implementing these changes over the next several Foreman releases. We’ll try to update here as we go along, and perhaps open additional RFCs if needed on specific parts.
Want to help?
There’s plenty of work to do! While there are a few core developers that are already planning to work on this, any help is appreciated. Please reach out to me if you want to help this effort at any stage. There will also be tasks that do not require coding skills, such as testing changes and updating documentation, so everyone is welcome to take part.