Roadmap Action - Multiple Hostgroups proposal

Support
1

Hi all,

At the Roadmap last week, I promised to put together a proposal to
satisfy Ohad's concerns around parameter conflicts when using multiple
hostgroups. So here it is.

This is the simplest implementation I can come with, which retains
some flexibility. There are, no doubt, implementations which will look
nicer in the UI, but would require more work. I'm aiming for
least-work-to-implement. Pretty can come later :stuck_out_tongue:

There's only one mock-up screenshot I've made in GIMP - I believe it
illustrates the key points about the changes to the Hostgroup type,
and is attached.

The concept is to extend the Hostgroup table, adding a numerical
column "Priority". This is a piece of integer data which describes
which Hostgroup should win when two Hostgroups are specifying the same
Parameter. In my example the lower value wins, but this could be
easily reversed if the community feels that's clearer. We can also
have some kind of validation - if you try to assign a second Hostgroup
of the same Priority to a given Host, the Save validation can throw an
error.

Let's see a use case:

In my screenshot, I could safely assign Base/Core/EMEA (which still
inherits Base and Core as before), and Base/Core/Build17 (which is
fine for now, as classes are singletons) which has a different
Priority, and also QA, which is different again. If I chose to assign
Base/Core/ENENA, this would not validate - this allows us to specify
mutually exclusive Hostgroups.

If QA and Base/Core/EMEA both specify parameter "foo", then QA will
win, and it's version will appear in the YAML output.

In terms of UI changes, I think we can re-use a lot of the code from
the Puppetclasses tab in the Host Edit page. We already parse a list
of classes and provide a tree structure - this would be fine for the
Hostgroups too, at least for now. For the Hostgroup Edit page, it's
just an extra editbox. Along with the DB change, it should just be
backend changes from there, mainly to the Save validation and the ENC
output code.

I'm happy to take a shot at coding this if there's agreement from the
community that this is a reasonable implementation. I know Ohad's got
a lot on his plate right now :stuck_out_tongue:

As always, feedback is not just welcome, it's actively encouraged :slight_smile:

Cheers,
Greg

multi_hostgroup.png1276×917 11.3 KB

2

I really like the idea of multiple host groups. Also the idea of
creating a validation to prevent someone from duplicating parameters
would likely be easier for use, but maybe not implementation.

Could do a validation check if the parameter already exists (by
name)… something like

new_parameter = user_parameter if Parameter.find(:all, :conditions =>
{ :name => user_parameter }, :limit => 1).empty?

I'm still new to Ruby so that's likely now how to best do it.

I do like the hierarchy that's present in 4.0, but at times I'd like
to give a server two roles that are at the same level, and can't
inherit from one another. For example I have "standard_linux/
web_server" and "standard_linux/mysql_server" , which at times exist
on the same server but not always.

A work around I've used is I created a module called "roles". So I
have classes like "mysql_server.pp" and "web_server.pp" that are
mirrors of what's defined in my Foreman hostgroups. Where multiple
host groups would really be useful is for things that contain the same
classes for all hosts using the group but not the same parameters.
For example I have a module for NUT (Network UPS Tools). Depending on
the server the virtual machines are on, or the data center they are in
I assign them different parameters in the hostgroup to correctly
configure the respective files. For now I use the the same "roles"
module, but my goal is to eventually do all Puppet administration via
Foreman (or as much as possible).

  • Trey
··· On Dec 12, 2:26 pm, Greg Sutcliffe wrote: > Hi all, > > At the Roadmap last week, I promised to put together a proposal to > satisfy Ohad's concerns around parameter conflicts when using multiple > hostgroups. So here it is. > > This is the simplest implementation I can come with, which retains > some flexibility. There are, no doubt, implementations which will look > nicer in the UI, but would require more work. I'm aiming for > least-work-to-implement. Pretty can come later :P > > There's only one mock-up screenshot I've made in GIMP - I believe it > illustrates the key points about the changes to the Hostgroup type, > and is attached. > > The concept is to extend the Hostgroup table, adding a numerical > column "Priority". This is a piece of integer data which describes > which Hostgroup should win when two Hostgroups are specifying the same > Parameter. In my example the lower value wins, but this could be > easily reversed if the community feels that's clearer. We can also > have some kind of validation - if you try to assign a second Hostgroup > of the same Priority to a given Host, the Save validation can throw an > error. > > Let's see a use case: > > In my screenshot, I could safely assign Base/Core/EMEA (which still > inherits Base and Core as before), and Base/Core/Build17 (which is > fine for now, as classes are singletons) which has a different > Priority, and also QA, which is different again. If I chose to assign > Base/Core/ENENA, this would not validate - this allows us to specify > mutually exclusive Hostgroups. > > If QA and Base/Core/EMEA both specify parameter "foo", then QA will > win, and it's version will appear in the YAML output. > > In terms of UI changes, I think we can re-use a lot of the code from > the Puppetclasses tab in the Host Edit page. We already parse a list > of classes and provide a tree structure - this would be fine for the > Hostgroups too, at least for now. For the Hostgroup Edit page, it's > just an extra editbox. Along with the DB change, it should just be > backend changes from there, mainly to the Save validation and the ENC > output code. > > I'm happy to take a shot at coding this if there's agreement from the > community that this is a reasonable implementation. I know Ohad's got > a lot on his plate right now :P > > As always, feedback is not just welcome, it's actively encouraged :) > > Cheers, > Greg > > multi_hostgroup.png > 79KViewDownload
3

> Hi all,
>
> At the Roadmap last week, I promised to put together a proposal to
> satisfy Ohad's concerns around parameter conflicts when using multiple
> hostgroups. So here it is.
>
> This is the simplest implementation I can come with, which retains
> some flexibility. There are, no doubt, implementations which will look
> nicer in the UI, but would require more work. I'm aiming for
> least-work-to-implement. Pretty can come later :stuck_out_tongue:
>
> There's only one mock-up screenshot I've made in GIMP - I believe it
> illustrates the key points about the changes to the Hostgroup type,
> and is attached.
>
> The concept is to extend the Hostgroup table, adding a numerical
> column "Priority".

Would that really work? I mean, what if you use the same hostgroup to
multiple hosts? (each time you might want some other group to win).

Thanks!
Ohad

··· On Mon, Dec 12, 2011 at 10:26 PM, Greg Sutcliffe wrote:

This is a piece of integer data which describes
which Hostgroup should win when two Hostgroups are specifying the same
Parameter. In my example the lower value wins, but this could be
easily reversed if the community feels that’s clearer. We can also
have some kind of validation - if you try to assign a second Hostgroup
of the same Priority to a given Host, the Save validation can throw an
error.

Let’s see a use case:

In my screenshot, I could safely assign Base/Core/EMEA (which still
inherits Base and Core as before), and Base/Core/Build17 (which is
fine for now, as classes are singletons) which has a different
Priority, and also QA, which is different again. If I chose to assign
Base/Core/ENENA, this would not validate - this allows us to specify
mutually exclusive Hostgroups.

If QA and Base/Core/EMEA both specify parameter “foo”, then QA will
win, and it’s version will appear in the YAML output.

In terms of UI changes, I think we can re-use a lot of the code from
the Puppetclasses tab in the Host Edit page. We already parse a list
of classes and provide a tree structure - this would be fine for the
Hostgroups too, at least for now. For the Hostgroup Edit page, it’s
just an extra editbox. Along with the DB change, it should just be
backend changes from there, mainly to the Save validation and the ENC
output code.

I’m happy to take a shot at coding this if there’s agreement from the
community that this is a reasonable implementation. I know Ohad’s got
a lot on his plate right now :stuck_out_tongue:

As always, feedback is not just welcome, it’s actively encouraged :slight_smile:

Cheers,
Greg

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

4

>> Hi all,
>>
>> At the Roadmap last week, I promised to put together a proposal to
>> satisfy Ohad's concerns around parameter conflicts when using multiple
>> hostgroups. So here it is.
>>
>> This is the simplest implementation I can come with, which retains
>> some flexibility. There are, no doubt, implementations which will look
>> nicer in the UI, but would require more work. I'm aiming for
>> least-work-to-implement. Pretty can come later :stuck_out_tongue:
>>
>> There's only one mock-up screenshot I've made in GIMP - I believe it
>> illustrates the key points about the changes to the Hostgroup type,
>> and is attached.
>>
>> The concept is to extend the Hostgroup table, adding a numerical
>> column "Priority".
>
> Would that really work? I mean, what if you use the same hostgroup to
> multiple hosts? (each time you might want some other group to win).
>

I believe hostgroups essentially map to a puppet a class.

$var1 = one
$var2 = two
class {'abc':}
class{''def':}

I do support the use case for multiple hostgroups so that something very common e.g
mysql can be added to other services as an addition.

But I believe they should just fail like puppet would fail if two such classes conflict on a variable or class name.

Adding a priority is essentially adding another orthogonal tree of classes which is confusing if nothing else.

··· On Dec 13, 2011, at 8:53 AM, Ohad Levy wrote: > On Mon, Dec 12, 2011 at 10:26 PM, Greg Sutcliffe > wrote:

Thanks!
Ohad

This is a piece of integer data which describes
which Hostgroup should win when two Hostgroups are specifying the same
Parameter. In my example the lower value wins, but this could be
easily reversed if the community feels that’s clearer. We can also
have some kind of validation - if you try to assign a second Hostgroup
of the same Priority to a given Host, the Save validation can throw an
error.

Let’s see a use case:

In my screenshot, I could safely assign Base/Core/EMEA (which still
inherits Base and Core as before), and Base/Core/Build17 (which is
fine for now, as classes are singletons) which has a different
Priority, and also QA, which is different again. If I chose to assign
Base/Core/ENENA, this would not validate - this allows us to specify
mutually exclusive Hostgroups.

If QA and Base/Core/EMEA both specify parameter “foo”, then QA will
win, and it’s version will appear in the YAML output.

In terms of UI changes, I think we can re-use a lot of the code from
the Puppetclasses tab in the Host Edit page. We already parse a list
of classes and provide a tree structure - this would be fine for the
Hostgroups too, at least for now. For the Hostgroup Edit page, it’s
just an extra editbox. Along with the DB change, it should just be
backend changes from there, mainly to the Save validation and the ENC
output code.

I’m happy to take a shot at coding this if there’s agreement from the
community that this is a reasonable implementation. I know Ohad’s got
a lot on his plate right now :stuck_out_tongue:

As always, feedback is not just welcome, it’s actively encouraged :slight_smile:

Cheers,
Greg

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

5

This was my first thought as well, I think setting a priority on that
level is going to be messy. Frankly, there's already too many places
you can prioritize and have things overriden and loose yourself
completely :).

Some environments will have their roles nicely cut out: They will have
a Bulky4UServer running their Database, a 2UServer running the Web,
and some 1UServer running the Application nodes. In this scenario,
Host Groups work perfectly fine, you won't ever need anything else.

Other environments might not have their roles so cleanly separated but
will mix and match many types of roles. In this scenario the Host
Group might end up meaning "Hardware Type", more or less, or you might
have different host groups more or less only to helt place machines in
different subnets etc.

Or, you could even use host groups as meaning "Cluster"… I.e. Host
Group "Cluster1" is a bunch of servers that might share something in
common.
As an example, they could be in the same backup group, meaning that
they all share the same backup storage server, or for other people it
might mean they're all placed in the same Rack thus the host groups
might be called "Rack1".

The solution treydock mention by using "roles" might end up being the
solution for people with servers who do multiple services or roles. I
also like to call it "collections".

Assuming using "collections" or "roles" in this manner is a good
enough solution, from my point of view there's one thing missing ->
transparency in the UI. If I were to hackishly solve this "problem"
now, I'd do the following:
I would decide that I create a module called "collection" or "role",
then have each collection/role submodule being called
"collection::Name". Then it'd be easy enough to hack the Foreman UI to
display all the Name parts of any collection:: modules being applied
to a certain host in all host lists and host overviews.

In this manner, I would see what roles a particular host have got
applied, without clicking "view all puppet classes" or editing the
puppet classes for a particular host.

Another way to do it could be to allow certain classes to be tagged,
and make all tagged classes end up being displayed in certain key UI
pages, such as the Host list and overview pages on a particular host.
It wouldn't make sense to display ALL modules/classes, because there
could be thousands of subclasses and unimportant classes which are in
Base, applied on all hosts anyway.

I've a c mock-up I just did here:
http://static.tmtowtdi.se/screenshots/screenshot.20111213_101448.png

I guess the Foreman way of solving it would mean adding Roles or
Collections support to Foreman, kindof like OP is suggesting here,
though I'm not sure I like it to be multiple Host Groups… that
sounds like it's going to be very messy!

··· On Tue, Dec 13, 2011 at 8:53 AM, Ohad Levy wrote: > On Mon, Dec 12, 2011 at 10:26 PM, Greg Sutcliffe > wrote: >> Hi all, >> >> At the Roadmap last week, I promised to put together a proposal to >> satisfy Ohad's concerns around parameter conflicts when using multiple >> hostgroups. So here it is. >> >> This is the simplest implementation I can come with, which retains >> some flexibility. There are, no doubt, implementations which will look >> nicer in the UI, but would require more work. I'm aiming for >> least-work-to-implement. Pretty can come later :P >> >> There's only one mock-up screenshot I've made in GIMP - I believe it >> illustrates the key points about the changes to the Hostgroup type, >> and is attached. >> >> The concept is to extend the Hostgroup table, adding a numerical >> column "Priority". > > Would that really work? I mean, what if you use the same hostgroup to > multiple hosts? (each time you might want some other group to win).

–
Mikael

6

>
>>> Hi all,
>>>
>>> At the Roadmap last week, I promised to put together a proposal to
>>> satisfy Ohad's concerns around parameter conflicts when using multiple
>>> hostgroups. So here it is.
>>>
>>> This is the simplest implementation I can come with, which retains
>>> some flexibility. There are, no doubt, implementations which will look
>>> nicer in the UI, but would require more work. I'm aiming for
>>> least-work-to-implement. Pretty can come later :stuck_out_tongue:
>>>
>>> There's only one mock-up screenshot I've made in GIMP - I believe it
>>> illustrates the key points about the changes to the Hostgroup type,
>>> and is attached.
>>>
>>> The concept is to extend the Hostgroup table, adding a numerical
>>> column "Priority".
>>
>> Would that really work? I mean, what if you use the same hostgroup to
>> multiple hosts? (each time you might want some other group to win).
>>
>
>
> I believe hostgroups essentially map to a puppet a class.
>
> $var1 = one
> $var2 = two
> class {'abc':}
> class{''def':}
>
> I do support the use case for multiple hostgroups so that something very common e.g
> mysql can be added to other services as an addition.
>
> But I believe they should just fail like puppet would fail if two such classes conflict on a variable or class name.

and what if you changed the group it self afterwards? should we check
all possible combinations of all host / hostgroups every time you
modify a hostgroup?

> Adding a priority is essentially adding another orthogonal tree of classes which is confusing if nothing else.
I agree

My gut feeling about this feature, that its more complex than it
sounds, and that having multiple hostgroups (via assignment) and
multiple hostgroups (via inheritance) is more trouble then its worth.

but as usually, I might be wrong :slight_smile:

Ohad

··· On Tue, Dec 13, 2011 at 11:26 AM, Steve Traylen wrote: > On Dec 13, 2011, at 8:53 AM, Ohad Levy wrote: >> On Mon, Dec 12, 2011 at 10:26 PM, Greg Sutcliffe >> wrote:

Thanks!
Ohad

This is a piece of integer data which describes
which Hostgroup should win when two Hostgroups are specifying the same
Parameter. In my example the lower value wins, but this could be
easily reversed if the community feels that’s clearer. We can also
have some kind of validation - if you try to assign a second Hostgroup
of the same Priority to a given Host, the Save validation can throw an
error.

Let’s see a use case:

In my screenshot, I could safely assign Base/Core/EMEA (which still
inherits Base and Core as before), and Base/Core/Build17 (which is
fine for now, as classes are singletons) which has a different
Priority, and also QA, which is different again. If I chose to assign
Base/Core/ENENA, this would not validate - this allows us to specify
mutually exclusive Hostgroups.

If QA and Base/Core/EMEA both specify parameter “foo”, then QA will
win, and it’s version will appear in the YAML output.

In terms of UI changes, I think we can re-use a lot of the code from
the Puppetclasses tab in the Host Edit page. We already parse a list
of classes and provide a tree structure - this would be fine for the
Hostgroups too, at least for now. For the Hostgroup Edit page, it’s
just an extra editbox. Along with the DB change, it should just be
backend changes from there, mainly to the Save validation and the ENC
output code.

I’m happy to take a shot at coding this if there’s agreement from the
community that this is a reasonable implementation. I know Ohad’s got
a lot on his plate right now :stuck_out_tongue:

As always, feedback is not just welcome, it’s actively encouraged :slight_smile:

Cheers,
Greg

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

7

Thanks for the feedback everyone :slight_smile:

Treydock / Mfridh - I kind of agree. I also use role based classes
right now. The problem I have is that I want to give lower levels of
support the ability to change things without giving them git access to
the puppet tree. I don't want them calling me up at 4am to change one
line in a puppet class - they should be able to alter than in Foreman.

Steve, I'm not sure I agree that it's an orthogonal set of classes. At
the simplest level, you're parsing a list of classes in each group,
and munging them together into a single list of classes in the output.
One hostgroup or two hostgroups - I still expect to return a single
list of classes to Puppet in the ENC data. However, confusion is
always to be avoided - perhaps we should make this optional via
More->Settings ?

Ohad, the Priority idea possibly not that great - I was trying to
avoid some pitfalls I'd seen in my other ideas :slight_smile: Perhaps I can re-use
an idea I wrote for my first ENC (before I moved to Foreman). Consider
the following example and psuedo-code:

  1. Re-use the the Puppetclasses tab for groups (or make them part of
    the same page - even better). Use this to allow a user to assign
    multiple hostgroups to a host.
  2. Don't sort the groups assigned to the host (unlike classes which
    are alphabetical)
  3. Loop over the groups in listed order. Once a parameter is set,
    further attempts to set it should be ignored
  4. (optional) Provide Up & Down arrows to sort the list. Initially,
    acceptable to remove & re-add groups to re-sort

That would allow different hosts to win in different cases. I think it
also avoids validation errors, since the YAML will ignore updated
parameters. We could work out ways to notify the user of those
conflicts later - I'm just trying to thrash out a prototype :slight_smile:

Closing comment on whether we should even be doing this - so long as
we make it optional, I don't see the harm. It's clearly a
controversial topic though, so I might make a branch on my github
account and see if I can make it fly - if I do, those interested can
have a play and give feedback.