Hostgroup re-design

Development
1

Ohad,

Amos and I are working on the hostgroup re-design. Here are the attributes that were are removing from hostgroups:

  • environment - determined by host environment
  • media - determined by host location
  • domain - determined by host location
  • subnet - determined by host location
  • architecture - determined by hardware profile
  • partition table - determined by OS template ???
  • root password - determined by OS template ???

Do you agree that partition table and root password can removed from hostgroup and put in OS template?

Hardware profile will include:

  • of cores

  • memory disk
  • disk size ???
  • architecture

We will add hardware profile to hostgroup. Do you agree?

Then, in order to provision a new host, you need to select:

  1. hostgroup & environment (WHAT)
  2. location & compute resource (WHERE)

Regards,

Joseph

2

> Amos and I are working on the hostgroup re-design. Here are the attributes that were are removing from hostgroups:

I'm not against refactoring hostgroups, but I do see some problems with
the proposal as-is. Please read my comments as constructive criticism
because that's how I intend it.

> * environment - determined by host environment

-1 because puppet classes are linked to an environment (and IMHO not
strong enough). Removing the environment would imply removing the puppet
classes as well, which would make configuring a bunch of hosts as a
group of webservers from foreman much harder. I can imagine people will
stop using the puppet class parameters and move to hiera to achieve the
role/profile pattern.

To give a better picture, let me describe how we now use host groups.

  • Make one per environment
    ** Select a set of defaults for environment, puppet master, puppet ca,
    often subnet, domain, OS, architecture, partition table as well
    (since we're pretty standardized on that).
    ** Add some base classes (base profile, such as SSH, nrpe, munin)
  • Create a sub-hostgroup per role
    ** Add the role specific classes (think database, webserver, application
    server, load balancer).

One option would be to add conditional puppet classes so that if you
select production, you get the production classes where in staging you
get the staging classes, but I am wondering if that wouldn't make it
very complicated.

Maybe the roles/profiles should be explicit. A puppet profile would be
tied to an environment, include puppet classes with optional parameter
defaults. Then a puppet role would be tied to an environment, include
some of the puppet profiles from that environment with some optional
parameter defaults. The role should also solve conflicts in the defaults
since multiple profile may include the same puppet classes with
different overrides.

Still, if you want to have a hostgroup per role, you either need to
include environment in the hostgroup, or make a puppet role group which
maps an environment to the environment specific classes with their
parameters.

> * media - determined by host location
> * domain - determined by host location
> * subnet - determined by host location
This assumes locations are enabled. What if one location has many
subnets and domains? Can a subnet belong to multiple locations?
> * architecture - determined by hardware profile
> * partition table - determined by OS template ???
Can you have multiple OS templates per OS? Such as LVM vs non-LVM or
ext4 vs xfs. Mutliple drives for your gluster fileserver etc. Maybe it
should be named OS profile?
> * root password - determined by OS template ???
>
> Do you agree that partition table and root password can removed from hostgroup and put in OS template?
>
> Hardware profile will include:

How will this work for bare metal servers? They do have an architecture,
but the others are hard to change.

> * # of cores
Some virtualisation allows you to select the number of cores per socket
and the amount of sockets. How will you deal with this?
> * memory disk
I assume you meant memory size.
> * disk size ???
I assume you meant disks with their respective sizes.
> * architecture

I'd add NICs with their respective names as well.

> We will add hardware profile to hostgroup. Do you agree?
>
> Then, in order to provision a new host, you need to select:
>
> 1) hostgroup & environment (WHAT)
> 2) location & compute resource (WHERE)

As should be clear by the length of questions, I highly value role as a
selection as well.

··· On Sun, Sep 29, 2013 at 09:16:04AM -0400, Joseph Magen wrote:
3

+1 disk size in hardware profile, really need this feature with ec2
instances, Foreman defaults to 8GB which is fine for app and web servers
but way too small when building DBs, Elasticsearch clusters etc.

Jim

··· On 29 September 2013 14:16, Joseph Magen wrote:

Ohad,

Amos and I are working on the hostgroup re-design. Here are the attributes
that were are removing from hostgroups:

  • environment - determined by host environment
  • media - determined by host location
  • domain - determined by host location
  • subnet - determined by host location
  • architecture - determined by hardware profile
  • partition table - determined by OS template ???
  • root password - determined by OS template ???

Do you agree that partition table and root password can removed from
hostgroup and put in OS template?

Hardware profile will include:

  • of cores

  • memory disk
  • disk size ???
  • architecture

We will add hardware profile to hostgroup. Do you agree?

Then, in order to provision a new host, you need to select:

  1. hostgroup & environment (WHAT)
  2. location & compute resource (WHERE)

Regards,

Joseph

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi Ewoud,

By removing the environment from hostgroup, we do not want to remove the puppet classes. I see the hostgroup as a template of defaults for a host. We could add a default environment to a hostgroup, but I thought it would be better to have each host created explicity assigned an environment. Based on the environment, the correct puppetclass is installed. For example, development environment has xyzclass-1.5-beta whereas production environment has 1.4-stable but it's the "same puppetclass" as defined in the hostgroup, just not environment specific. It's just an idea, but maybe it wont work, since we need an environment to know the available puppetclasses

Regarding hardware profile, the plan is to enter the minimum requirements, so when when running discovery it can find the bare metal servers that are appropriate for provisioning a particular server for hostgroup / HW profile.

Regards,

Joseph

··· ----- Original Message ----- From: "Ewoud Kohl van Wijngaarden" To: "foreman-dev" Sent: Sunday, September 29, 2013 6:16:39 PM Subject: Re: [foreman-dev] Hostgroup re-design

On Sun, Sep 29, 2013 at 09:16:04AM -0400, Joseph Magen wrote:

Amos and I are working on the hostgroup re-design. Here are the attributes that were are removing from hostgroups:

I’m not against refactoring hostgroups, but I do see some problems with
the proposal as-is. Please read my comments as constructive criticism
because that’s how I intend it.

  • environment - determined by host environment

-1 because puppet classes are linked to an environment (and IMHO not
strong enough). Removing the environment would imply removing the puppet
classes as well, which would make configuring a bunch of hosts as a
group of webservers from foreman much harder. I can imagine people will
stop using the puppet class parameters and move to hiera to achieve the
role/profile pattern.

To give a better picture, let me describe how we now use host groups.

  • Make one per environment
    ** Select a set of defaults for environment, puppet master, puppet ca,
    often subnet, domain, OS, architecture, partition table as well
    (since we’re pretty standardized on that).
    ** Add some base classes (base profile, such as SSH, nrpe, munin)
  • Create a sub-hostgroup per role
    ** Add the role specific classes (think database, webserver, application
    server, load balancer).

One option would be to add conditional puppet classes so that if you
select production, you get the production classes where in staging you
get the staging classes, but I am wondering if that wouldn’t make it
very complicated.

Maybe the roles/profiles should be explicit. A puppet profile would be
tied to an environment, include puppet classes with optional parameter
defaults. Then a puppet role would be tied to an environment, include
some of the puppet profiles from that environment with some optional
parameter defaults. The role should also solve conflicts in the defaults
since multiple profile may include the same puppet classes with
different overrides.

Still, if you want to have a hostgroup per role, you either need to
include environment in the hostgroup, or make a puppet role group which
maps an environment to the environment specific classes with their
parameters.

  • media - determined by host location
  • domain - determined by host location
  • subnet - determined by host location
    This assumes locations are enabled. What if one location has many
    subnets and domains? Can a subnet belong to multiple locations?
  • architecture - determined by hardware profile
  • partition table - determined by OS template ???
    Can you have multiple OS templates per OS? Such as LVM vs non-LVM or
    ext4 vs xfs. Mutliple drives for your gluster fileserver etc. Maybe it
    should be named OS profile?
  • root password - determined by OS template ???

Do you agree that partition table and root password can removed from hostgroup and put in OS template?

Hardware profile will include:

How will this work for bare metal servers? They do have an architecture,
but the others are hard to change.

  • of cores

Some virtualisation allows you to select the number of cores per socket
and the amount of sockets. How will you deal with this?

  • memory disk
    I assume you meant memory size.
  • disk size ???
    I assume you meant disks with their respective sizes.
  • architecture

I’d add NICs with their respective names as well.

We will add hardware profile to hostgroup. Do you agree?

Then, in order to provision a new host, you need to select:

  1. hostgroup & environment (WHAT)
  2. location & compute resource (WHERE)

As should be clear by the length of questions, I highly value role as a
selection as well.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I would like environment removed from hostgroups our VPCs for devint,
functest, nft and production are internally identical, I would like to
simply assign a hostgroup defining say "webservers" rather than needing to
define 4 or more hostgroup that are to all intents and purposes identical.

Jim

··· On 30 September 2013 07:40, Joseph Magen wrote:

Hi Ewoud,

By removing the environment from hostgroup, we do not want to remove the
puppet classes. I see the hostgroup as a template of defaults for a host.
We could add a default environment to a hostgroup, but I thought it would
be better to have each host created explicity assigned an environment.
Based on the environment, the correct puppetclass is installed. For
example, development environment has xyzclass-1.5-beta whereas production
environment has 1.4-stable but it’s the “same puppetclass” as defined in
the hostgroup, just not environment specific. It’s just an idea, but maybe
it wont work, since we need an environment to know the available
puppetclasses

Regarding hardware profile, the plan is to enter the minimum requirements,
so when when running discovery it can find the bare metal servers that are
appropriate for provisioning a particular server for hostgroup / HW profile.

Regards,

Joseph

----- Original Message -----
From: “Ewoud Kohl van Wijngaarden” ewoud@kohlvanwijngaarden.nl
To: “foreman-dev” foreman-dev@googlegroups.com
Sent: Sunday, September 29, 2013 6:16:39 PM
Subject: Re: [foreman-dev] Hostgroup re-design

On Sun, Sep 29, 2013 at 09:16:04AM -0400, Joseph Magen wrote:

Amos and I are working on the hostgroup re-design. Here are the
attributes that were are removing from hostgroups:

I’m not against refactoring hostgroups, but I do see some problems with
the proposal as-is. Please read my comments as constructive criticism
because that’s how I intend it.

  • environment - determined by host environment

-1 because puppet classes are linked to an environment (and IMHO not
strong enough). Removing the environment would imply removing the puppet
classes as well, which would make configuring a bunch of hosts as a
group of webservers from foreman much harder. I can imagine people will
stop using the puppet class parameters and move to hiera to achieve the
role/profile pattern.

To give a better picture, let me describe how we now use host groups.

  • Make one per environment
    ** Select a set of defaults for environment, puppet master, puppet ca,
    often subnet, domain, OS, architecture, partition table as well
    (since we’re pretty standardized on that).
    ** Add some base classes (base profile, such as SSH, nrpe, munin)
  • Create a sub-hostgroup per role
    ** Add the role specific classes (think database, webserver, application
    server, load balancer).

One option would be to add conditional puppet classes so that if you
select production, you get the production classes where in staging you
get the staging classes, but I am wondering if that wouldn’t make it
very complicated.

Maybe the roles/profiles should be explicit. A puppet profile would be
tied to an environment, include puppet classes with optional parameter
defaults. Then a puppet role would be tied to an environment, include
some of the puppet profiles from that environment with some optional
parameter defaults. The role should also solve conflicts in the defaults
since multiple profile may include the same puppet classes with
different overrides.

Still, if you want to have a hostgroup per role, you either need to
include environment in the hostgroup, or make a puppet role group which
maps an environment to the environment specific classes with their
parameters.

  • media - determined by host location
  • domain - determined by host location
  • subnet - determined by host location
    This assumes locations are enabled. What if one location has many
    subnets and domains? Can a subnet belong to multiple locations?
  • architecture - determined by hardware profile
  • partition table - determined by OS template ???
    Can you have multiple OS templates per OS? Such as LVM vs non-LVM or
    ext4 vs xfs. Mutliple drives for your gluster fileserver etc. Maybe it
    should be named OS profile?
  • root password - determined by OS template ???

Do you agree that partition table and root password can removed from
hostgroup and put in OS template?

Hardware profile will include:

How will this work for bare metal servers? They do have an architecture,
but the others are hard to change.

  • of cores

Some virtualisation allows you to select the number of cores per socket
and the amount of sockets. How will you deal with this?

  • memory disk
    I assume you meant memory size.
  • disk size ???
    I assume you meant disks with their respective sizes.
  • architecture

I’d add NICs with their respective names as well.

We will add hardware profile to hostgroup. Do you agree?

Then, in order to provision a new host, you need to select:

  1. hostgroup & environment (WHAT)
  2. location & compute resource (WHERE)

As should be clear by the length of questions, I highly value role as a
selection as well.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

>
> From: "Ewoud Kohl van Wijngaarden" <ewoud@kohlvanwijngaarden.nl>
>
>> * environment - determined by host environment
>
> -1 because puppet classes are linked to an environment (and IMHO not
> strong enough). Removing the environment would imply removing the puppet
> classes as well, which would make configuring a bunch of hosts as a
> group of webservers from foreman much harder. I can imagine people will
> stop using the puppet class parameters and move to hiera to achieve the
> role/profile pattern.

It's a tricky thing. The environment specified in the Hostgroup is only a
default, which can be changed when you create a Host. Given that different
classes may exist in EnvA compared with EnvB, the classes in the Hostgroup
may no longer make sense.

However, if we drop environments from Hostgroups, we also drop the ability
to filter out classes that users of other environments shouldn't see/use.
It's not an improvement to be able to add classes from EnvB to your host if
you're not permitted to see EnvB (i.e a developer should not see/use
classes from Production).

Perhaps we could specify a list of allowable environments for that
Hostgroup, and then present a list of classes from all of those envs? I do
feel specifying a 'default' environment is a good idea, especially as its
so easily changed when creating a new Host.

> I would like environment removed from hostgroups our VPCs for devint,
> functest, nft and production are internally identical, I would like to
> simply assign a hostgroup defining say "webservers" rather than needing to
> define 4 or more hostgroup that are to all intents and purposes identical.
>

See above. In addition, if they are identical, why not use one hostgroup
and change the environment field when creating a Host?

>
> Regarding hardware profile, the plan is to enter the minimum requirements,
>> so when when running discovery it can find the bare metal servers that are
>> appropriate for provisioning a particular server for hostgroup / HW profile
>>
>
I echo Ewould on the concerns about multiple subnets/domains etc. However,
if all we are doing is moving the place to specify "defaults" (which is
what Hostgroups do) then this should be fine. The concern about forcing the
use of Locations is valid however - do we have a plan that doesn't involve
Locations?

I'm also not sure it makes sense to base a big refactor on an optional
plugin. While it's very thoughtful of you to use Discovery as the
cornerstone of the design, I'm wondering what the usecases are for people
without Discovery and without Locations.

For me, the natural place for a lot of this data is the Compute Reosurce -
they naturally tend to default to a certain Subnet, Domain, and Media. The
choice of OS has always been little more than a suggestion, and could just
as easily be on a CR as on a Hostgroup. The only slight concern is how to
handle bare metal - but we've talked about treating that as a form of CR in
the past anyway, so this may be a good time to make that change?

Greg

··· On 30 September 2013 10:07, James Bailey wrote:
7

> From: "Ewoud Kohl van Wijngaarden" <ewoud@kohlvanwijngaarden.nl>
>>
>>> * environment - determined by host environment
>>
>> -1 because puppet classes are linked to an environment (and IMHO not
>> strong enough). Removing the environment would imply removing the puppet
>> classes as well, which would make configuring a bunch of hosts as a
>> group of webservers from foreman much harder. I can imagine people will
>> stop using the puppet class parameters and move to hiera to achieve the
>> role/profile pattern.
>>
>
> It's a tricky thing. The environment specified in the Hostgroup is only a
> default, which can be changed when you create a Host. Given that different
> classes may exist in EnvA compared with EnvB, the classes in the Hostgroup
> may no longer make sense.
>
> However, if we drop environments from Hostgroups, we also drop the ability
> to filter out classes that users of other environments shouldn't see/use.
> It's not an improvement to be able to add classes from EnvB to your host if
> you're not permitted to see EnvB (i.e a developer should not see/use
> classes from Production).
>
> Perhaps we could specify a list of allowable environments for that
> Hostgroup, and then present a list of classes from all of those envs? I do
> feel specifying a 'default' environment is a good idea, especially as its
> so easily changed when creating a new Host.
>

Does it make sense to break the hostgroup <> puppetclass relationships and
move it to a new 'policy' based object (which has many classes/envs)?

this would allow us to further talk about policy's, and later bind them to
a system hostgroup/role/template?

we then could extend this object and talk about allowed classes in given
envs etc?
this should hopefully reduce the duplication between hostgroups with regard
to classes?

I don't have a strong opinion about this (yet), but we could include
multiple policies in one hostgroup?

>
>> I would like environment removed from hostgroups our VPCs for devint,
>> functest, nft and production are internally identical, I would like to
>> simply assign a hostgroup defining say "webservers" rather than needing to
>> define 4 or more hostgroup that are to all intents and purposes identical.
>>
>
> See above. In addition, if they are identical, why not use one hostgroup
> and change the environment field when creating a Host?
>
>>
>> Regarding hardware profile, the plan is to enter the minimum
>>> requirements, so when when running discovery it can find the bare metal
>>> servers that are appropriate for provisioning a particular server for
>>> hostgroup / HW profile
>>>
>>
> I echo Ewould on the concerns about multiple subnets/domains etc. However,
> if all we are doing is moving the place to specify "defaults" (which is
> what Hostgroups do) then this should be fine. The concern about forcing the
> use of Locations is valid however - do we have a plan that doesn't involve
> Locations?
>

I think that each CR resources should be mapped to foreman resources (e.g.
this VPC is this subnet), if locations are not enabled, well… then you
don't have filtering ?

> I'm also not sure it makes sense to base a big refactor on an optional
> plugin. While it's very thoughtful of you to use Discovery as the
> cornerstone of the design, I'm wondering what the usecases are for people
> without Discovery and without Locations.
>

I think that we still want a hardware profile, that would work across all
CR (e.g. you define one time CPU, mem etc and you match it automatically
based on the CR), if you want similar functionality on BM, you add the
discovery plugin… i think that's acceptable.

··· On Wed, Oct 2, 2013 at 1:58 PM, Greg Sutcliffe wrote: > On 30 September 2013 10:07, James Bailey wrote:

For me, the natural place for a lot of this data is the Compute Reosurce -
they naturally tend to default to a certain Subnet, Domain, and Media. The
choice of OS has always been little more than a suggestion, and could just
as easily be on a CR as on a Hostgroup. The only slight concern is how to
handle bare metal - but we’ve talked about treating that as a form of CR in
the past anyway, so this may be a good time to make that change?

Greg

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Just yesterday I opened a feature request on this. Great to hear discussion
going on around redesign of hostgroups!

Right now, we have a set of scripts that provision an environment. This has
disk, cpu, memory, network standards defined per-hostgroup. When we
provision a new host via the Foreman API we aupply all these pieces. It
would be nice to see the option of having these settings happen at the
hostgroup level, with the option of having defaults set at the parent
hostgroup level (we use parent/child hostgroups quite a bit).

This would allow us to build an entire environment with sane defaults set
for the function. For example, a CI environment could have bare-bones
settings to make for a functional environment, whereas a perf environment
has the same settings that match production.

Another option would be to create a virtual hardware profile, which could
then be associated with a hostgroup. This would help to reduce complexity
in managing separate virtual hardware settings in each hostgroup.

Also, on another topic that Joseph mentioned… we have a single OS
template across our entire environment, and we like being able to specify
partition tables at the hostgroup level.

Thanks for everyone's contributions - this is a really valuable tool for
our organization.

Greg

9

> Does it make sense to break the hostgroup <> puppetclass relationships and
> move it to a new 'policy' based object (which has many classes/envs)?
>
> this would allow us to further talk about policy's, and later bind them to
> a system hostgroup/role/template?
>

I need to have a think about that, and read Ewoud's post again, but it
could make sense. Over-complicating the UI would be my biggest concern I
think…

> I think that each CR resources should be mapped to foreman resources (e.g.
>> this VPC is this subnet), if locations are not enabled, well… then you
>> don't have filtering ?
>>
>

> For me, the natural place for a lot of this data is the Compute Reosurce -
> they naturally tend to default to a certain Subnet, Domain, and Media. The
> choice of OS has always been little more than a suggestion, and could just
> as easily be on a CR as on a Hostgroup. The only slight concern is how to
> handle bare metal - but we've talked about treating that as a form of CR in
> the past anyway, so this may be a good time to make that change?

My Libirt CR has multiple Subnets - but broadly I agree. The things which
are set as "defaults" in Hostgroups today should really be set as defaults
on the CR. Bare metal is the tricky one to set a default for though - that
can have many subnets/domians/etc.

··· On 2 October 2013 12:09, Ohad Levy wrote:
10

In https://groups.google.com/d/msg/foreman-dev/M9_mGo1lGl8/A-sA5ntJh1cJ
I suggested something similar. It may also make the puppet integration a
bit more optional. I'm unsure if you still want to assign classes to
hosts or just profiles.

··· On Wed, Oct 02, 2013 at 01:09:01PM +0200, Ohad Levy wrote: > On Wed, Oct 2, 2013 at 1:58 PM, Greg Sutcliffe wrote: > > > From: "Ewoud Kohl van Wijngaarden" > >> > >>> * environment - determined by host environment > >> > >> -1 because puppet classes are linked to an environment (and IMHO not > >> strong enough). Removing the environment would imply removing the puppet > >> classes as well, which would make configuring a bunch of hosts as a > >> group of webservers from foreman much harder. I can imagine people will > >> stop using the puppet class parameters and move to hiera to achieve the > >> role/profile pattern. > >> > > > > It's a tricky thing. The environment specified in the Hostgroup is only a > > default, which can be changed when you create a Host. Given that different > > classes may exist in EnvA compared with EnvB, the classes in the Hostgroup > > may no longer make sense. > > > > However, if we drop environments from Hostgroups, we also drop the ability > > to filter out classes that users of other environments shouldn't see/use. > > It's not an improvement to be able to add classes from EnvB to your host if > > you're not permitted to see EnvB (i.e a developer should not see/use > > classes from Production). > > > > Perhaps we could specify a list of allowable environments for that > > Hostgroup, and then present a list of classes from all of those envs? I do > > feel specifying a 'default' environment is a good idea, especially as its > > so easily changed when creating a new Host. > > > > Does it make sense to break the hostgroup <> puppetclass relationships and > move it to a new 'policy' based object (which has many classes/envs)? > > this would allow us to further talk about policy's, and later bind them to > a system hostgroup/role/template? > > we then could extend this object and talk about allowed classes in given > envs etc? > this should hopefully reduce the duplication between hostgroups with regard > to classes? > > I don't have a strong opinion about this (yet), but we could include > multiple policies in one hostgroup?