Removing host groups from Foreman

Development
1

Hey,

on the OSAD conference we slightly discussed hostgroups. I strongly believe hostgroups are over-engineered and utterly complex for users to understand and operate and for developers to keep the code on par when we do changes. I believe major problems are inheritance, nesting and the fact they appear to set everything for hosts but that’s not the case. This leads to confusion on both sides - users do not understand exactly what is set by hostgroup and what is not (Additional Information, Compute Resource, Compute Profile, Host Collection and Content Host). Developers have the same problem and when working on features and bugs this ultimately leads to more issues, even for new features (I remember that PXE loader needed like 3-5 fixes until it was hostgroup-friendly).

An extreme solution to this problem is completely get rid of hostgroups. Perhaps there can be something in the middle, but I want to stop and think about it - what if we dropped them completely? Here are some questions for folks:

Do you use hostgroups?

What are the features you like?

What features you don’t like or don’t use?

How would you approach this problem? Different concept? More fields or less fields in hostgroup?

2

Hi,
in my opinion, host groups are needed to create “New Host” templates, i.e. to fill out all the required fields automatically if you want to deploy a new host using the GUI. Thus the user is able to do a kind of one-klick deployment.
Removing this functionality would make new host creation a mess.
-Mark

3

My opinion is rather extreme here: Let’s remove compute profiles first (they’re worse than hostgroups), hostgroups next.

I think we should still allow this, hostgroups and compute profiles aren’t a good solution for this though.
I’d like to have two personas: an admin that configures templates or blueprints. And a user (that can be an admin) that can deploy hosts based on this blueprints. The idea is to separate the “how to deploy” configuration from the “what to deploy”.
In addition, I really love labels or tags. This should be similar to github labels. An admin can configure labels with a name and a color (“type/production” - red, “type/testing” - blue). These labels would be available via ENC and can be used for filtering and searching. But they would not define any parameters.
We can think about introducing folders for structuring or to define parameters. But all the other attributes should be defined in a blueprint.

Just my two cents.

2 Likes
4

Personally I use host groups to set parameters and rely on the inheritance.

That makes me think about config groups, but I must admit I’ve never used those. Perhaps we could leverage those but I think that currently hosts can have multiple config groups so you have the problem of conflict resolution when multiple groups define the same parameter.

5

I also had a crazy idea this morning. Maybe it’s not yet complete.
We could split the host model up into a central model containing only the name of the host (maybe a mac address), and move everything else a host can have or be (like an operating system, a configuration environment, a content view, generic parameters, pxe configuration, a compute resource, …) into separate optional facets.
Each of those facets should then be matched by a model for specific configuration templates (like hostgroups are used today). An instance of this template should know which values of the host facet should be set to a certain value (like operatingsystem or domain), which should be autogenerated (like ipaddress) and which should be asked from the ui (like hostname). Maybe add constraints, choices or suggestions here.
A certain collection of those configuration templates would replace the hostgroup concept. And from the knowledge which fields were marked to be asked by the admin, a wizard can be composed. Also, if for example the content facet configuration template is not even selected in the configuration template set, no questions about it are asked at all, and no corresponding host facet is created.
From the users perspective, one would select a configuration template, fill in the 3-5 questions not yet preconfigured, without being able to change the ones one is not supposed to and hit submit.

6

We would not remove the capability of setting flags from a “template” of some kind. But it would not needed be too difficult to understand I think.

7

How much of a problem this would be when we converted this to flat-like structure?

8

Fully agree.

9

In Foreman’s Foreman instance we have a structure:

  • Base
  • Base/Builders
  • Base/Builders/Debian
  • Base/Builders/Red Hat
  • Base/Foreman
  • Base/Jenkins
  • Base/Redmine
  • Base/Web

However, it appears we mostly use it to assign classes and the only parameter we set is for REX rather than a Puppet class. Those are usually overridden for specific hosts rather than host groups.

10

This kinda confirms my thought that people usually use nesting up to 3 levels which should not be huge deal to convert to flat structure. Of course, still with references to “parent” host records, so when needed users would be able to change something let’s say in “Base” context and Foreman would (optionally) propagate (copy) the change to all of its “children”.

11

Do you use hostgroups?

Yes, all our hosts are provisioned and managed via hostgroups (and we also use compute profiles on hostgroup :slight_smile: )

What are the features you like?

  • inheritance. We use it to manage our config standards, and make users provision host with 3 or 4 selections in interface (organization, location, hostgroup, network subnet and that’s it). There is a lot of things to think about when creating host and hostgroup + inheritance permit to our users to make less errors on host creation. inheritance permit us to apply common puppet classes (or config groups) on a “base” hostgroup and then add some specific classes on lower groups level. We also apply activation keys on hostgroup basis. If we want to add/remove one param or puppet class to some group of hosts, it’s easy because we do it in an unique place. It also help us to organize our hosts and some other applications configuration rely on hostgroups (eg: We use prometheus to do some monitoring and we generate prometheus configuration based on our hostgroups to select which exporters to scrap. it permit us to monitor correctly and automaticaly new hosts after provisioning). I don’t see how to make it easily without inheritance (but any advices are welcome on this part).

example hostgroup hierarchy:

base/servers/el7/appserverX/appY
base/servers/el7/appserverX/appY
base/servers/el7/databases/mariadb
base/servers/el7/ovirt/nodes
base/servers/el7/ovirt/engines
base/servers/el8/
...
  • smart class parameter matcher on hostgroups (mostly all of our puppet config are matchers on hostgroups, and we use inheritance on it too)

What features you don’t like or don’t use?

Sorry but I think we use mostly all features of hostgroups :slight_smile:

12

Hey Lukas,

thanks for the chat at OSAD.

Do you use hostgroups?

Yes

What are the features you like?

  • Hostgroups in general

Gives us the possibility to create templates in a way that a minimum set of values have to be entered for deployment(Name,Organization, Location and HostGroup). Thus, the actual deployment of new host is possible for everybody. No detailed knowledge of items like target cluster, networking, storage,vcenter, or OS Minor Version or anything else is required. This also ensures a high level of standardization and orchestration for each required level. We use for our BareMetal installation AutoDeploy Rules that also depend on Hostgroups settings.

  • Inheritance

Our configuration ended up in more than 70 Hostgroups with a level of max 5 nested. So Inheritance is an important feature for us. OS Minor Version, Subnet Changes, New Parameter… is minimized to 3-4 Changes…

  • Parameter Settings within hostgroups

Our hostgroups are configured by a data center administrator in cooperation with the actual target department. Any other employee is enabled to create new hosts based on these templates without any need of deep infrastructure knowledge. For us, it enables automated integration with applications and clusters as well as integrating extensions for load balancer environments. The parameters are further evaluated in our Kickstart/Snippet templates which enables us to use a single Kickstart file for many different scenarios.

  • Taxonomy on hostgroups

Not every user can access all hostgroups. This allows us, that the teams can only deploy hosts following a preconfigured template in order to keep the standards upright.

Our hierarchy of hostgroups is designed like that:
OS/ApplicationType/Environment(Prod/Stage/Dev)/[detailed purpose/location ]

e.g.:

CentOS6/PHPApp/Prod/behind_LB/phpapp-websng/
CentOS7/Integration/dev
CentOS7/Integration/prod
CentOS7/Integration/stage
OracleLinux/DB/DNFS/Stage/RZ1
OracleLinux/DB/DNFS/Stage/RZ2
OracleLinux/PostgreSQL/Prod
Windows/2012R2
Windows/2012R2/Prod
Windows/2012R2/Test
ESX6.5/CITRIX/RZ1
ESX6.5/VM_PROD_LINUX
RedHat7/SAP

What features you don’t like or use?

We do not use puppet class configuration inside the host templates. Instead, we use PuppetFacts within our Kickstart templates (PuppetRole, Clustername,….) configured by Parameter within the hostgroup. The rest is triggered by the Puppet Server itself. From our point of view, this is a good option to avoid problems of understandings as changes will not be applied immediately to all hosts as opposed to managing them within foreman.

How would you approach this problem? Different concept? More fields or less fields in hostgroup?

In order to get rid of problems of understandings, an easy approach would be to mark all fields accordingly (e.g. by selecting a special input color or whatever) or let the user do an additional commit (All changes will be applied to running hosts immediately – you you want to proceed?) or something similar…

In general I think it is never good for a general acceptance to discontinue features rather than hiding them from the first view. Even if alternatives might be interesting, for a large group of users, the world outside is more complicated, environments are complicated and becoming more and more complex everyday demanding a solution that does not only work with kind of simple to follow but also allowing for a high level of flexibility.

Susanne

2 Likes
13

Susanne, thanks for a deep insight into your deployment.

We would not just remove host groups, this thread is a discussion about a different approach to achieve the same goal. Removing hostgroups without providing similar mechanism was never on the table.

That is a decent number indeed, if hostgroups would be replaced with a flat structure this would lead to 70 “templates” or “blueprints” or whatever this was called. Not any increase here.

How do you approach the fact that when you change something in a hostgroup, you effectively change the value for all hosts which use the hostgroup but haven’t redefined the value? The major issue here is that you don’t clearly see which hosts have redefined the value. You can easily miss some.

We will absolutely keep parameters, these are essential to both provisioning and cfg management.

The same, taxonomy is not going anywhere (although I could imagine it being more simple).

We often hear that.

A change is always a change, disturbing. We need to be sure if that’s good enough. These are just discussions, we are all quite busy so I don’t expect these to be implemented anytime soon. But it’s good to ask now and gather feedback. Also there would be transition period for some time so users can prepare.

Thanks!

14

we rely heavily on hostgroups combined with locations,

we have different staging envs dev -> int -> test -> prod, all this stages have their own locations in foreman and only the specific hostgroup for the stage (=location) is mapped to the location.

so in dev, you can only select the hostgroups for dev and the hostgroup provides the rest of the information to deploy via 1 klick btw. fully automated via hammer cli

idk if i would like if hostgroups would be removed

15

We are still getting to know Foreman and Puppet, so not very representative, but here goes. We use host groups mostly to select and parameterize puppet classes that we build in-house (mostly wrappers to known modules and some custom things). In fact we discovered Foreman while learning about the Puppet ecosystem, and since we weren’t using server provisioning yet, we killed two birds with one stone.

So besides provisioning we are using Foreman as our main Puppet configurator. Not sure if it’s the best idea, and we’re already suffering with the fact that the latest Foreman stopped supporting the remote execution plugin in Debian 9, which we used for “Puppet Run”. (if any dev wants to look into this, examples are here and here.)

If Config Groups could hold Puppet class parameters (which would make a lot of sense given their name!), I guess we could live without host groups, but we’re trying to make good use of them in general including inheritance, which seems an important tool to keep our sanity as the infrastructure grows.