A small group of installer maintainers have been talking recently about changes we’d like to make to the installation experience. This post aims to outline those changes to open up the conversation and get some community feedback on the ideas. We plan to follow up this posting with a weekly, open to the community, meeting where we will discuss the feedback and heartier points in real time. This will include related topics such as certificate redesign.
@ekohl @sureshkumar_thirugna @wclark please keep me honest here
1) Move answers files to be treated like a database
The journey for a parameter provided by the installer flows through stages of validation within the installer hooks and Puppet. There are stages of the installer that can manipulate those parameters are configure others. In fact, the installer, and underlying Kafo framework, treat parameters as inputs that are manipulated in memory, then combined with the answers file from disk which are then committed back to disk. In this way, the installer can be thought of performing a database transaction where the answers are the database schema. In some cases, users modify the answers files directly rather than using input parameters to the installer which can subvert the workflow.
The proposal is to elevate the answers file to be treated like a database. The answers file would move to
/var/lib/foreman-installer rather than
/etc to help drive this notion home. Direct editing of the answers file would be unsupported. A new feature would be added that allows supplying an input file to the installer with yaml formatted options the user would like to supply. This would continue to support the configuration-as-code desire for some users.
An aspect of this would be to add some tooling to give users a clear view of the current landscape that is configured through human readable output of the answers and config.
2) Single answers and config file per installation
Early on in Kafo and installer life, scenario support was added. This included the theoritical idea to switch between scenarios. Since that time, only three scenarios have ever existed with no true way to switch between them. Reducing to a single “scenario” per installtaion reduces complexity and allows for a more standard experience. That is we can have the following per installation:
- single, same named answers file (answers.yaml)
- single, same named config file (config.yaml)
- single installer log (installer.log)
- single error log (error.log)
This plays along nicely with the idea of moving answers to be more like a database. This also evolves into the idea of removing “scenarios” as we know them today to be more like groups of input parameters that achieve a defined set of features enabled rather than a inflexible scenario. Scenarios could instead be thought of as grouping parameters to turn features on as inputs to the installer rather than hard and fast rules. This allows thinking more in terms of a base setup with additional functionality layered on top per a users desire. This view of scenarios will require some of the Katello on Foreman issues to handled for Katello being added (every other plugin could be handled this way today).
3) Actionable concrete error handling and logging
Understanding why the installer failed at different stages can range from easy to nearly impossible. The most reported case is when the smart proxy registration fails due to an error with the server. The user must investigate multiple logs to figure out what the error is and where to resolve it. The installer documentation, and testing code often default to using
-v to work around the poor base output. Further, the scenarios are configured to output debug level logging by default. This results in using these advanced features as a crutch for proper output for normal operation of the installer. There are improvements that can be made in this area:
- Show progress reporting at all stages of installer #29241
- Better error handling and reporting by smart proxy when communicating with Foreman
- Cleaner error reporting when the installer errors at hooks and during puppet apply
- Default to info logging for stdout (debug for logs) #29749
- Add explicit error log #29748
- Formatting of error messages #29750
- Remove generic catch-all error messages in installer
4) Manageable sets of and sizing of available parameters
The installer notoriously presents the user with a mountain of parameters to configure as the default. One solution has been to move some parameters within the puppet modules into an ‘advanced’ grouping. This can be taken further to group related parameters and hide more parameters as advanced to not overwhelm the users.
- Can optionally group parameters together towards a feature set
E.g. postgres parameter
- Higher level parameters for enabling features that require multiple parameters to be orchestrated
5) Drop --no-enable
For every module, Kafo presents a
--enable and a
--no-enable option. However, the in most cases once a module, and its associated functionality, is enabled there is no reverse path. For example, there is no true way to remove a Foreman plugin, so disabling it via the installer provides a false sense to the user. Until we can officially support this, we should drop the presentation of
--no-enable to reduce user confusion and potential for users to wind up in a bad state.
There are areas increased testing workflows could reduce churn for developers within the installer ecosystem and broader Foreman developer community. Installer changes impact the flow of changes through nightly and development environmnets. The areas in a priortized order of testing to tackle:
- Hook unit testing
- Move common functionality (see our boot/01-helpers) to kafo to enable easier unit testing and out of the hooks
- Migration unit testing
- Drop --disable-system-checks in CI testing
- Idempotency testing
- Run the installer and re-run the installer to ensure nothing has changed
- Can add a bats test and use detailed exit codes
- Add integration test to installer PRs
- Ability to generate an RPM with a command on a PR [generate rpm]
- Automatic update to Puppetfile based on a puppet module specified in PR
- Certificate replacement CI testing
- Installation scenario variation testing, e.g.
- remote execution
7) DHCP and DNS Implementations
DNS and DHCP implementations aren’t great, users should be steered to better implementaions.
There is a lack of clarity for users as to which parameters are for the Foreman Proxy DNS module and for ISC bind. This becomes especially confusing when you use a different provider. An option is to move them to top level modules just like Puppet so you get
--dns-* options while
--foreman-proxy-dns-* options are only for the Foreman Proxy module. The same is true for DHCP.
8) foreman-maintain’s role in installer workflows
The foreman-maintain project sits in an odd place within our ecosystem at present. The project houses a lot of maintainenace activities such as backup, restore, service handling and provides the ability to orcehstrate workflow upgrades for a scenario. The workflows in foreman-maintain can orchestrate foreman-installer but also provide value to foreman-installer (e.g. ability to disable know services for a given installation) providing a slight chicken-egg relationship. Further, foreman-maintain is only used by a subset of the ecosystem (partially Katello, and largely Satellite) and is not packaged everywhere.
Fundamentally, we need to answer the question: is foreman-maintain a core tool within the ecosystem? If so, we need to work to make it available everywhere, documented into the base documentation and ensure we’ve well defined its boundaries. Some further points to discuss:
- Elevate foreman-maintain as a first class citizen with a clear boundary between it and the installer
- Increase testing upstream
- Dropping the “upgrade” notion from the installer, let foreman-maintain orchestrate
- Package foreman-maintain for Debian
A high-level list of features provided from the README:
- health checks
- upgrade orchestration
- service handling
- content upgrades (for Pulp 2 to Pulp 3)