Modularizing Foreman - High-level features

Development RFCs
1

One of the discussions held at the Brno meetup in May revolved around merging some common plugins into Core, as well as making some existing core functionality optional.
One example of such existing functionality are Puppet, which is currently always present and displayed to users even if they don’t use Foreman for configuration management or use a different tool. Another example is provisioning. While not all users use Foreman for provisioning management, there are multiple resources and fields related to it which are always displayed. Some of them can be partially hidden by using the “unattended” setting, but this setting is relatively confusing and not well documented or tested.
If some of the existing plugins are merged into core, we will see more cases such as this - of functionality that clutters the UI and complicates workflows but isn’t being used by all users.

One of the ideas that came up in the discussion was creating some mechanism for allowing our users to define which high-level features they want to use, and than have the UI show only the relevant fields and resources. This will hopefully also help simplify workflows and code paths a bit by providing a “happy path” for each such feature that will be tested properly.
Ideally, it would also be possible to enable these at runtime using the GUI and asking the user for needed information, making the initial install simpler.
Another thought was to auto-detect which such features are enabled by looking at the resources defined (e.g. smart proxy features, compute resources, etc.) but this path may lead to a discoverability problem as users would have to first define all the prerequisits to unlock their workflows instead of selecting the workflow and requesting the required information/configuration.
It might also be good to enable plugins to define such feature or patch into existing ones.

Before we start working on implementing this, it would be good to hear from the community if this sounds like a good direction and worth the effort.
I would also be happy to hear from @Roxanne_Hoover and @terezanovotna regarding the usability aspect of this effort with respect to UX best practices.

Finally, we had some initial thoughts regarding what would be considered such “High level feature” (we could also use a better name), but there are likely others we missed - the main idea here is that we don’t want this to turn into a big mess of options that make things more complicated than they are. Some such features we were thinking of:

  • Provisioning - possibly split into bare metal and virtual?
  • Puppet
  • Ansible
  • Salt
  • Chef
  • Remote execution

Please comment below with further suggestions (or even just to say this sounds good to you).

2 Likes
2

Initially it feels like a good idea, but I have some concerns about the implementation.

What I always liked about Foreman is that it treated all provision as equal. At a previous employer I’ve also used “bare metal” on VMware because we didn’t have access to the VMware API. This may be confusing to the user. Ideally we could even do the same operations (power management etc) on bare metal using IPMI to get an equivalent experience. I don’t have good answers to this.

There’s a few concepts that are easy to keep in because they’re shared with other types. These are reports and facts.

We then have a few items in the main navigation:

  • Environments
  • Classes
  • Config Groups
  • Smart Variables
  • Smart Class Parameters

First of all: Smart Variables should go. This was useful before we had Smart Class Parameters and typed global variables (introduced in 1.22).

Second, the Config Groups could be extracted to a more common level. There’s actually 2 PRs open to add this information to the Foreman Ansible Inventory:

https://github.com/ansible/ansible/pull/55508

I would say the same thing is roughly true for environments: it can be useful in Ansible as well. The implementation is not that much more than adding a label to a host. In the database it’s environment but in places it’s starting to be renamed to Puppet Environment. I’d like to see this be Configuration Management Environment.

Puppet Classes could be become a tab on the Configuration Management Environment. The same goes for Smart Class Parameters. Essentially there’s a hierarchy Environment → Puppet Class → Smart Class Parameter. I think a well designed page would have sufficient power to use that.

Then you’re down to one top level menu item. If you don’t use configuration management, then you can just ignore that.

Next up is the host form. Today we already hide the Puppet Master and Puppet CA selection if there’s no proxy with such a feature. We could apply the same thing to the Puppet Classes tab: if you don’t select an environment, hide the tab. This would be similar to how we hide/show the VM tab depending on the Deploy on (bare metal/virt) selection.

I’ve also been thinking about a Smart Proxy Capability called ENC which indicates the proxy is capable of assigning classes. If it doesn’t, then you don’t need to show classes and assign classes to hosts. This may be useful in case you want to provision a host with Puppet but use another system (Puppet Enterprise, Hiera) to manage the content.

Then there’s also some graphs and details on the Host Detail and Dashboard. These need a serious do-over regardless of this feature. Those graphs on the Host Detail page don’t feel very useful to me.

Now a lot of this can probably be applied to Ansible, Salt and Chef as well.

In summary, I think that a well designed UI can have a navigation that removes a lot of the need for this.

1 Like
3

Personally I do like the concept of making Foreman more mix-and-match kind of setup, so a :+1: from me.

In my own opinion, our current notion of a plugin is a good candidate for feature definition. I would try to avoid yet another classification system, since it can complicate user’s choices. For example if you have a plugin that spans multiple “high-level features”, like puppet that exposes both reporting and config management, it would be difficult to correlate between the two things.

If I take an example from atom, which has a lot of plugins too - they only have a notion of a plugin, and each plugin can influence multiple aspects/high-level features of the system.

1 Like
4

Unpopular opinion: I don’t think we need to do this. I see few major problems which can be solved separately without need of any “feature turn on/off” capability:

  • New/Edit Host form and this one is being reworked from the ground up. If we refactor this form into a wizard, then we can hide things “on the fly” and there is no need of removing things from the UI. Most of the users I believe have no problem with many items in the main menu - if they don’t use it, they don’t click it.
  • Host detail form is a blast from the past, it needs to be redesigned shortly after the form I think. But again, we can take a different approach than some global “on/off” settings - redesigining it in a modular way. What I like is our Dashboard and if we allow our users to do the same with host detail pages (e.g. compose your own “view” of your hosts) then we put enough tools into users hands which can solve the problem.
  • Deployment without Puppet is something that we constantly improve release after release and the ultimate goal is to be able to install Foreman without any Puppet bits like servers, certificates or other tooling.

What else is a major problem we are trying to solve here? We just got rid of taxonomy and unattended settings because it was causing pains. I believe that we should try hard to avoid the same. I like the idea of merging some plugins into core, but if we do it in an unintrusive way there should not be any need for the “feature” feature.

5

Personally I’m very happy to see this. Foreman is a great framework to achieve infrastructure management tasks of many types. Not every user needs all of them. It should be easy to adopt new types of tasks, but it shouldn’t clutter the UI and leave users confused by the complexity of the system. Therefore I think enabling/disabling of parts is important. The only mechanism we have today is plugins, but that is not enough. We need support in core that defines the feature and provide a way for plugin to implement or extend it. It would help us to better describe use cases, structure the documentation, improve isolation of components, let users only work with parts, that are relevant for them.

Also it would make it possible to merge more functionality from plugins to core and organize it altogether into features - e.g. no discovery or bootdisk being second class citizens, people are either interested in provisioning or not. Feature itself can be more granular inside, but as a whole, this is functionality users either want to disable. Then we can talk about whether we need to support 5 ways of donig the PXE based provisioning.

Ideally, it would also be possible to enable these at runtime using the GUI and asking the user for needed information, making the initial install simpler.

This would be a big benefit IMHO. The experience of adding more things to the app today is not great. Removal is impossible for most of plugins.

Before we start working on implementing this, it would be good to hear from the community if this sounds like a good direction and worth the effort.

+1 from me

I would also be happy to hear from @Roxanne_Hoover and @terezanovotna regarding the usability aspect of this effort with respect to UX best practices.

do you have more specific questions? that would probably help

Finally, we had some initial thoughts regarding what would be considered such “High level feature” (we could also use a better name), but there are likely others we missed - the main idea here is that we don’t want this to turn into a big mess of options that make things more complicated than they are. Some such features we were thinking of:

I totally agree with we don’t want to have hunders of options for this. 10 main features seems like the most. Sadly “feature” is already used term on proxy so could be a bit confusing. Other words that could describe these - “Area”, “Use case”, “Gear”, “App” but it’s not the most important things atm :slight_smile:

  • Provisioning - possibly split into bare metal and virtual?

I agree with @ekohl here, I know users doing “baremetal” flow on in virtualized environments too so I’d keep it together.

Puppet, Ansible, Salt, Chef

Separation per cfgmgmt tool seems reasonable, though beware today foreman_ansible can be used for both rex and cfgmgt and potentially in future for tower integration. I wonder if we should split this into more features. E.g. cfgmgmt may enable environments or ENC button.

Maybe we can break this down a bit differently. E.g.

  • reporting (config reports, facts, trends, statistics) - this is where we consume and display data provided from various external tools
  • configuration management (puppet classes, variables, parameters, environemnts and equivalents for other than puppet tools) - this is where we actively talk to cfgmgmt systems to perform the configuration
  • core would remain supporting tools such as (audits, report templates, tasks, host and grouping, user and grouping, authn, authz, parameters - the general one, taxonomy, bookmarks).

If this works with things in core, later I’d be happy to have the same ability for plugins. E.g. openscap plugin would benefit from being enabled/disabled easily, even though we probably want to keep it out of the core, simply because it feels very specific and well isolated use case.

6

I think there are good things we can do today like this one. But if we apply it everywhere, imagine you’re mainly interested in katello plugin usecase, or openscap plugin use case. Then you want Foreman to provide just the platform functionality - rbac, authentication, audits, logging, perhaps tasks tooling, maybe reports, but you don’t want to ever mess with cfgmgmt or provisioning. Why would you have such menu items?

Next up is the host form. Today we already hide the Puppet Master and Puppet CA selection if there’s no proxy with such a feature. We could apply the same thing to the Puppet Classes tab: if you don’t select an environment, hide the tab. This would be similar to how we hide/show the VM tab depending on the Deploy on (bare metal/virt) selection.

I think you are in fact suggesting the same thing but driven not by explicit enabling/disabling of the feature but by detecting what proxies provide you. My concern with this approach is, it’s a bit more complicated with multitenancy and permissions. Should users see puppet menu if they can’t see any puppet enabled proxy? Should a feature x be enabled if a proxy lives in one of current organization suborganization? Also the experience for user then is, go to foreman proxy and enable the feature there (probably means running the installer, like today for any other plugin). Also there’s functionality that is not handled through proxy (compute resource, compute profiles), how would we detect availability of this?

I like your suggestion of consolidating cfgmgmt menu items under one, I’d still like to have ability to had that item explicitly by disabling the (new) feature, not by disabling the feature on proxy.

7

I in fact think that it would help formalize the use cases that Foreman focuses on. Plugins today add stuff randomly and spans accross multiple areas of the project. I’d be happy to use the feature concept to formalize parts of Foreman :slight_smile:

I think it was you who said the menu is crazy and you never know which group you need to navigate to find item x :slight_smile:

if you deploy without puppet, why do you want to keep it in UI? I’m not talking about reports and facts, I’m mostly interested in smart class params, smart variables, environments, puppet classes

This is entirely different granularity level. Think of 5-10 features vs 100 random settings you mentioned. To some degree we even have these ares today on settings page, e.g. provisioning tab. Over time we can see the settings would deserve a reorg.

8

I would be happy if we ask ourselves

  • What is the user trying to do? e.g. what is her task? (e.g. provision a host, patch a machine, roll out something etc)
  • what workflows would match the above question, e.g. as a user I would like to patch, I would need to filter based on X then do Y and then trigger something

then, imho, this should lead into a new set of wire frames that are more task oriented, which IMHO should fall into one of the buckets we talked about above (provisioning, configuration, remote executing etc).

1 Like
9

That makes me wonder: if the feature is disabled, should also not allow any smart proxy to register a certain feature? That way you can get the best of both worlds: disabling globally removes certain menu items and disallows proxy feature registration. Since no proxy is available, you automatically disable certain functionality elsewhere meaning a much smaller code difference.

Code difference was the major reason to drop the taxonomy/login settings. It would make testing easier.

10

I think this is the more generalized version of what I tried to describe about the Puppet class workflow: a more focused UI/UX would remove a lot of need for this altogether.

11

We only support one PXE workflow, you can choose PXE loader but the process is the same.

No, I said that our menu is badly structured - almost everything related to provisioning is underneath Host and the rest seem to be not utilized. I was suggesting to restructure it.

More simple codebase. Do not think about it as a hidden menu item - it will not be that simple. Think about consequences and upgrade scenarios. What happens when user hide certain parts and then there will be some blocking bug preventing from doing main task (like edit an existing host)? These ideas come to my mind.

All I want to say is that we need to justify this effort, it will be challenging and all users will experience it. I haven’t seen enough arguments why we need to do it this way. I think we can improve our UI in such a way that navigation and main workflows will be usable without explicitly hiding anything.

What we are gonna do not to repeat our mistakes with unattended flag? I have concerns about testing this.

Here is a relevant thought: We should not be hiding things just because (we think) user don’t want to see it. We should advertise what’s relevant instead.

12

Are you suggesting breaking the plugins into multiple ones, based on project area? e.g. foreman_puppet_reporting and foreman_puppet_configuration instead of a single puppet plugin.