Installation Experience Design Discussion

Development
1

A small group of installer maintainers have been talking recently about changes we’d like to make to the installation experience. This post aims to outline those changes to open up the conversation and get some community feedback on the ideas. We plan to follow up this posting with a weekly, open to the community, meeting where we will discuss the feedback and heartier points in real time. This will include related topics such as certificate redesign.

@ekohl @sureshkumar_thirugna @wclark please keep me honest here

1) Move answers files to be treated like a database

The journey for a parameter provided by the installer flows through stages of validation within the installer hooks and Puppet. There are stages of the installer that can manipulate those parameters are configure others. In fact, the installer, and underlying Kafo framework, treat parameters as inputs that are manipulated in memory, then combined with the answers file from disk which are then committed back to disk. In this way, the installer can be thought of performing a database transaction where the answers are the database schema. In some cases, users modify the answers files directly rather than using input parameters to the installer which can subvert the workflow.

The proposal is to elevate the answers file to be treated like a database. The answers file would move to /var/lib/foreman-installer rather than /etc to help drive this notion home. Direct editing of the answers file would be unsupported. A new feature would be added that allows supplying an input file to the installer with yaml formatted options the user would like to supply. This would continue to support the configuration-as-code desire for some users.

An aspect of this would be to add some tooling to give users a clear view of the current landscape that is configured through human readable output of the answers and config.

2) Single answers and config file per installation

Early on in Kafo and installer life, scenario support was added. This included the theoritical idea to switch between scenarios. Since that time, only three scenarios have ever existed with no true way to switch between them. Reducing to a single “scenario” per installtaion reduces complexity and allows for a more standard experience. That is we can have the following per installation:

  • single, same named answers file (answers.yaml)
  • single, same named config file (config.yaml)
  • single installer log (installer.log)
  • single error log (error.log)

This plays along nicely with the idea of moving answers to be more like a database. This also evolves into the idea of removing “scenarios” as we know them today to be more like groups of input parameters that achieve a defined set of features enabled rather than a inflexible scenario. Scenarios could instead be thought of as grouping parameters to turn features on as inputs to the installer rather than hard and fast rules. This allows thinking more in terms of a base setup with additional functionality layered on top per a users desire. This view of scenarios will require some of the Katello on Foreman issues to handled for Katello being added (every other plugin could be handled this way today).

3) Actionable concrete error handling and logging

Understanding why the installer failed at different stages can range from easy to nearly impossible. The most reported case is when the smart proxy registration fails due to an error with the server. The user must investigate multiple logs to figure out what the error is and where to resolve it. The installer documentation, and testing code often default to using -v to work around the poor base output. Further, the scenarios are configured to output debug level logging by default. This results in using these advanced features as a crutch for proper output for normal operation of the installer. There are improvements that can be made in this area:

  • Show progress reporting at all stages of installer #29241
  • Better error handling and reporting by smart proxy when communicating with Foreman
  • Cleaner error reporting when the installer errors at hooks and during puppet apply
  • Default to info logging for stdout (debug for logs) #29749
  • Add explicit error log #29748
  • Formatting of error messages #29750
  • Remove generic catch-all error messages in installer

4) Manageable sets of and sizing of available parameters

The installer notoriously presents the user with a mountain of parameters to configure as the default. One solution has been to move some parameters within the puppet modules into an ‘advanced’ grouping. This can be taken further to group related parameters and hide more parameters as advanced to not overwhelm the users.

  • Basic
  • Advanced
  • Can optionally group parameters together towards a feature set
    E.g. postgres parameter
  • Higher level parameters for enabling features that require multiple parameters to be orchestrated

5) Drop --no-enable

For every module, Kafo presents a --enable and a --no-enable option. However, the in most cases once a module, and its associated functionality, is enabled there is no reverse path. For example, there is no true way to remove a Foreman plugin, so disabling it via the installer provides a false sense to the user. Until we can officially support this, we should drop the presentation of --no-enable to reduce user confusion and potential for users to wind up in a bad state.

6) Testing

There are areas increased testing workflows could reduce churn for developers within the installer ecosystem and broader Foreman developer community. Installer changes impact the flow of changes through nightly and development environmnets. The areas in a priortized order of testing to tackle:

  • Hook unit testing
  • Move common functionality (see our boot/01-helpers) to kafo to enable easier unit testing and out of the hooks
  • Migration unit testing
  • Drop --disable-system-checks in CI testing
  • Idempotency testing
    • Run the installer and re-run the installer to ensure nothing has changed
    • Can add a bats test and use detailed exit codes
  • Add integration test to installer PRs
    • Ability to generate an RPM with a command on a PR [generate rpm]
    • Automatic update to Puppetfile based on a puppet module specified in PR
  • Certificate replacement CI testing
  • Installation scenario variation testing, e.g.
  • remote execution
  • dns
  • puppet
  • ansible

7) DHCP and DNS Implementations

DNS and DHCP implementations aren’t great, users should be steered to better implementaions.
There is a lack of clarity for users as to which parameters are for the Foreman Proxy DNS module and for ISC bind. This becomes especially confusing when you use a different provider. An option is to move them to top level modules just like Puppet so you get --dns-* options while --foreman-proxy-dns-* options are only for the Foreman Proxy module. The same is true for DHCP.

8) foreman-maintain’s role in installer workflows

The foreman-maintain project sits in an odd place within our ecosystem at present. The project houses a lot of maintainenace activities such as backup, restore, service handling and provides the ability to orcehstrate workflow upgrades for a scenario. The workflows in foreman-maintain can orchestrate foreman-installer but also provide value to foreman-installer (e.g. ability to disable know services for a given installation) providing a slight chicken-egg relationship. Further, foreman-maintain is only used by a subset of the ecosystem (partially Katello, and largely Satellite) and is not packaged everywhere.

Fundamentally, we need to answer the question: is foreman-maintain a core tool within the ecosystem? If so, we need to work to make it available everywhere, documented into the base documentation and ensure we’ve well defined its boundaries. Some further points to discuss:

  • Elevate foreman-maintain as a first class citizen with a clear boundary between it and the installer
  • Increase testing upstream
  • Dropping the “upgrade” notion from the installer, let foreman-maintain orchestrate
  • Package foreman-maintain for Debian

A high-level list of features provided from the README:

  • health checks
  • upgrade orchestration
  • service handling
  • backup
  • restore
  • maintenance-mode
  • content upgrades (for Pulp 2 to Pulp 3)
2 Likes
2

I like it in general, but have some notes to add:

If possible I would like to keep this supported. It is not a big need, but I have done some test to production migration by copying and adjusting the answerfile and running the installer afterwards which was a very easy way to do so.

Not all of them can be dropped, some are used for good like --no-enable-foreman in a Puppet server or Smart proxy standalone installation. If there was a scenario for this ok, but at the current state having to know them or finding them in the documentation if they are not presented would be also a bad experience. But yes, dropping the useless ones would remove some clutter.

While not great, they really help in demo or even simple and green-field scenarios. I have also used them quite often with --noop --dont-save-answers to test run changes needed on already existing installations.

1 Like
3

This is the Red Hat definition of unsupported: it may work (and at least I have no intention of changing the file format in a big way), but you when you break it, you get to keep both pieces. Admins are used to editing files in /etc and we want to set the expectations right by placing it in /var/lib/foreman-installer.

I agree this was kind of a bad phrasing for a point I’ve tried to make. There are cases where they don’t make sense.

For example, in the Katello scenario the --no-enable-katello and --no-enable-foreman don’t make a lot of sense and you will end up in a broken use case. These options only confuse.

I’m also unhappy about --no-enable-*-plugin-* since it stops managing them, but they remain installed. This is a bad user experience. I’ve been thinking about enable/disable hooks as a solution to this. For example:

on_class_enable 'foreman::plugin::myplugin' do
  require_class 'foreman'
end

on_class_disable 'foreman::plugin::myplugin' do
  reject_disable_if_active
end

This would check that foreman is enabled when foreman::plugin::myplugin is enabled and show a clear error to the user. On disable it would check the answers file and if the module was previously enabled, it’s not possible to disable it. The user is presented with a clear error. This probably requires some override to force it, in case the user manually removed the plugin package and database side for example.

What I never liked was the --foreman-proxy-dns-* options which were vague. It was very unclear what was for the Foreman Proxy DNS module and for ISC bind. This becomes especially confusing when you use a different provider. I’ve been thinking about moving them to top level modules just like Puppet so you get --dns-* options while --foreman-proxy-dns-* options are only for the Foreman Proxy module. The same is true for DHCP.

On a related note, I also missed the options to do domain management. While ISC bind doesn’t allow this, other providers have APIs for this. With bind we can extend the Foreman Proxy module to create these files as well if we really want to. This would move away from the whole scenario where the installer sets up one magical domain and into a scenario where it’s all done at runtime in Foreman itself. This was my inspiration for the capabilities framework in Foreman Proxy but I never got around to do the actual domain implementation.

DHCP is more complicated because ISC dhcpd doesn’t start if there is no subnet and I don’t want to get into the space of setting up dhcp proxies for other VLANs etc so this probably remains an installer thing for some time.

1 Like
4

I’m quite enthusiastic about everything in this proposal! Much of what has been proposed here would have made my first install attempt much easier and less frustrating.

Both points 2 and 4, at least the quoted sections above, seem very similar to me. I’m very much in support of the idea of grouping parameters together in such a way that they clearly implement a given feature and make it obvious which parts of the feature are required vs optional. When I was doing my first install, I found I had to go through a process of trial and error to work out what was required vs optional, in order to add/configure a given feature.

With the proposed move to a YAML input file, much of this grouping and feature enablement could be done just with good annotation of a full featured example file and allowing the user to uncomment desired sections.

If these installer changes were also matched with use case oriented scenarios in the documentation, that would be a huge step forward for new users.

I’ve been really impressed with the suggestions in recent RFCs, the changes made on the documentation front and the decisions that went into the design of version 2. I like the direction in which this project is heading.

1 Like
5

This Thursday the plan is to discuss this RFC and related ones as part of the community demo. This is intended to be an open for questions and discussion session to help refine these designs. Please join!