The life-cycle of a puppet module within foreman

Hey
Really enjoying foreman and now we're looking at putting more structure
around our puppet workflow.
I have a question about how to handle puppet modules in hostgroups when
moving them between environments. I'd also be interested to hear how you
guys work with your modules and if you had any best practice in this area.
We want to push our modules through the standard workflow of dev, test and
prod. If we add a module into a hostgroup that spans all three
environments, while the module only exists in our dev environment puppet
runs on servers in test and prod start failing because they can't find the
module.
Is there anyway around this or a better way to work?
All the solutions we've thought up feel pretty inelegant

  • Creating empty modules in dev, test and prod initially
  • Manually adding the module to all the appropriate dev and/or test
    servers then remembering to remove them when you do finally make it to
    production
  • Creating separate hostgroups for each environment and then remembering
    to keep them in sync

I love to hear that we're doing it wrong and get some suggestions about
better workflows but it would be nice if foreman could be aware of what
modules are in what environments (as it already is) and intelligently put
the blinkers over puppet so servers in production don't error out because
it expects a module that isn't ready yet.

Thanks,
Charlie

> Hey
>
> Really enjoying foreman and now we're looking at putting more structure around our puppet workflow.
> I have a question about how to handle puppet modules in hostgroups when moving them between environments. I'd also be interested to hear how you guys work with your modules and if you had any best practice in this area.
> We want to push our modules through the standard workflow of dev, test and prod. If we add a module into a hostgroup that spans all three environments, while the module only exists in our dev environment puppet runs on servers in test and prod start failing because they can't find the module.
> Is there anyway around this or a better way to work?
> All the solutions we've thought up feel pretty inelegant
> • Creating empty modules in dev, test and prod initially
> • Manually adding the module to all the appropriate dev and/or test servers then remembering to remove them when you do finally make it to production
> • Creating separate hostgroups for each environment and then remembering to keep them in sync
> I love to hear that we're doing it wrong and get some suggestions about better workflows but it would be nice if foreman could be aware of what modules are in what environments (as it already is) and intelligently put the blinkers over puppet so servers in production don't error out because it expects a module that isn't ready yet.

··· On Dec 7, 2012, at 7:41 PM, Charlie Derwent wrote: ---- I tend to have only minimal hostgroups for development and assign the modules to individual development nodes to test before I migrate them to production and add them to production 'hostgroups'. I identify these hostgroups by adding '-dev' to the production name as in 'base' (production) and 'base-dev' (development).

I don’t know of any more comprehensive way to do things.

Craig

Nothing's perfect, and you're right that all the workarounds are
fairly clumsy. This one is my preference though - it means that you
don't get any unexpected surprises, everything is there for inspection
in the git log.

You can make it slightly more elegant by having a common modules path.
Suppose your puppet.conf is:

modulepath = /etc/puppet/environments/$environment/modules:/etc/puppet/common-modules

You can then drop an empty module into common-modules and it
immediately exists for all environments. Then you're free to import a
real module into each env as you roll forward.

Greg

··· On 8 December 2012 02:41, Charlie Derwent wrote: > Hey > > Creating empty modules in dev, test and prod initially

> Hey
> Really enjoying foreman and now we're looking at putting more structure
> around our puppet workflow.
> I have a question about how to handle puppet modules in hostgroups when
> moving them between environments. I'd also be interested to hear how you
> guys work with your modules and if you had any best practice in this area.
> We want to push our modules through the standard workflow of dev, test
> and prod. If we add a module into a hostgroup that spans all three
> environments, while the module only exists in our dev environment puppet
> runs on servers in test and prod start failing because they can't find the
> module.
> Is there anyway around this or a better way to work?
> All the solutions we've thought up feel pretty inelegant
>
> - Creating empty modules in dev, test and prod initially
> - Manually adding the module to all the appropriate dev and/or test
> servers then remembering to remove them when you do finally make it to
> production
> - Creating separate hostgroups for each environment and then
> remembering to keep them in sync
>
> I love to hear that we're doing it wrong and get some suggestions about
> better workflows but it would be nice if foreman could be aware of what
> modules are in what environments (as it already is) and intelligently put
> the blinkers over puppet so servers in production don't error out because
> it expects a module that isn't ready yet.
>
>

Hi Charlie,
We have been looking at this recently and I'm fairly sure we will in fact
stop including classes via foreman. We will contine to use
forman for specifying hostgroups but the mapping from there to manifests
will be static via site.pp. e.g a host in hostgroup

a/b/c/d

will load the classes (if they exist)

class{a:}
class{'a::b':}
class{'a::b::c':}
class{'a::b::c::'d':}

This puts more in the git repository which we can then version as you
describe.

For more info I'm half through writing a doc now: http://cern.ch/go/pr9D

··· On Saturday, 8 December 2012 03:41:17 UTC+1, Charlie Derwent wrote:

Thanks,
Charlie

>
>> Hey
>> Really enjoying foreman and now we're looking at putting more structure
>> around our puppet workflow.
>> I have a question about how to handle puppet modules in hostgroups when
>> moving them between environments. I'd also be interested to hear how you
>> guys work with your modules and if you had any best practice in this area.
>> We want to push our modules through the standard workflow of dev, test
>> and prod. If we add a module into a hostgroup that spans all three
>> environments, while the module only exists in our dev environment puppet
>> runs on servers in test and prod start failing because they can't find the
>> module.
>> Is there anyway around this or a better way to work?
>> All the solutions we've thought up feel pretty inelegant
>>
>> - Creating empty modules in dev, test and prod initially
>> - Manually adding the module to all the appropriate dev and/or test
>> servers then remembering to remove them when you do finally make it to
>> production
>> - Creating separate hostgroups for each environment and then
>> remembering to keep them in sync
>>
>> I love to hear that we're doing it wrong and get some suggestions about
>> better workflows but it would be nice if foreman could be aware of what
>> modules are in what environments (as it already is) and intelligently put
>> the blinkers over puppet so servers in production don't error out because
>> it expects a module that isn't ready yet.
>>
>>
>
> Hi Charlie,
> We have been looking at this recently and I'm fairly sure we will in
> fact stop including classes via foreman. We will contine to use
> forman for specifying hostgroups but the mapping from there to manifests
> will be static via site.pp. e.g a host in hostgroup
>
> a/b/c/d
>
> will load the classes (if they exist)
>
> class{a:}
> class{'a::b':}
> class{'a::b::c':}
> class{'a::b::c::'d':}
>
> This puts more in the git repository which we can then version as you
> describe.
>
> For more info I'm half through writing a doc now: http://cern.ch/go/pr9D
>

hmm… what about param classes? if you do that, you would lose the ability
to override the value inforeman?

secondly, what can we do better in foreman? is it simply the audit which is
missing?

lets reopen this for discussion!

thanks!
Ohad

··· On Mon, Dec 17, 2012 at 3:47 PM, Steve Traylen wrote: > On Saturday, 8 December 2012 03:41:17 UTC+1, Charlie Derwent wrote:

Thanks,
Charlie


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/foreman-users/-/fq7uaPAz2xgJ.

To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to
foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/foreman-users?hl=en.

I've reviewed your document, and you did find some important issues.

  1. a hostgroup does not contain a list of classes per environment
    this is correct, and I'm wondering what should be the right handling here,
    i see a couple of options.
    a. keep the list as is.
    b. manage a puppet class list per env.
    c. make sure the suggested classes are part of the selected env before
    returning the class list.

which one would you prefer? each one has its pro and cons, my preference
would be option C, its trivial to implement and would require the least
amount of maintenance, however, there is still a chance of a conflict (e.g
out of date imported classes etc).

option b. might require a lot of work, maybe it can be done with some
ui enchantments.

  1. dividing which classes are defined in foreman, vs. which are included
    via a manifest.
    I guess this is a fine line, where a design decision needs to be taken, I
    personal would not want to expose any internal class (e.g. app::config) or
    a infrastructure module (e.g. module that is only included by another
    module.

I normally filter them out on my foreman instance
(see config/ignored_environments.yml.sample for details).

I guess at the end, it depends who uses foreman (if its the same guy who
does the puppet manifests or rather someone else who has no idea about the
manifests and just want to consume it as a service.

I would eventually select classes that users might want to override their
value (via plain parameters or class parameters), and let all of the rest
to be managed via puppet.

Feedback?

thanks,
Ohad

··· On Mon, Dec 17, 2012 at 3:47 PM, Steve Traylen wrote:

For more info I’m half through writing a doc now: http://cern.ch/go/pr9D

>
> For more info I'm half through writing a doc now: http://cern.ch/go/pr9D
>
> I've reviewed your document, and you did find some important issues.
>
> 1. a hostgroup does not contain a list of classes per environment
> this is correct, and I'm wondering what should be the right handling here, i see a couple of options.
> a. keep the list as is.
> b. manage a puppet class list per env.
> c. make sure the suggested classes are part of the selected env before returning the class list.
>
> which one would you prefer? each one has its pro and cons, my preference would be option C, its trivial to implement and would require the least amount of maintenance, however, there is still a chance of a conflict (e.g out of date imported classes etc).
>
> option b. might require a lot of work, maybe it can be done with some ui enchantments.
>

C definitely makes sense , there's a horrible case where you add a class X every where and it breaks every environment of nodes not
containing that class.

As for B maybe at some places but the list will not be in step with the git files, i.e when I do 'git merge myfeature' into production I might also want the class list to be in step. I certainly want to
be testing the same class as I then have in production.

> 2. dividing which classes are defined in foreman, vs. which are included via a manifest.
> I guess this is a fine line, where a design decision needs to be taken, I personal would not want to expose any internal class (e.g. app::config) or a infrastructure module (e.g. module that is only included by another module.
>
> I normally filter them out on my foreman instance (see config/ignored_environments.yml.sample for details).
>

I was not aware of the filter (would be good to add to rpm), that makes a lot of sense, my doc did not
mention but the explosion of classes was certainly unattractive with the web interface but not a main reason for us checking.

>
> I guess at the end, it depends who uses foreman (if its the same guy who does the puppet manifests or rather someone else who has no idea about the manifests and just want to consume it as a service.
>

git branches and environments work incredibly well for isolating your configurations. So anything
outside that same mechanism is then a mismatch against that.

> I would eventually select classes that users might want to override their value (via plain parameters or class parameters), and let all of the rest to be managed via puppet.
>
The tricky ones are e.g apache::vhost{'myservice': template => 'foot':} which describes a whole service perfectly well and could go in a site.pp just like that.

··· On Dec 17, 2012, at 3:28 PM, Ohad Levy wrote: > On Mon, Dec 17, 2012 at 3:47 PM, Steve Traylen wrote:

Feedback?

thanks,
Ohad


You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

Thanks for all the input into this question, it's nice to see the team and
I are moving in the right direction. I'd have to agree with Ohad and Steve
and say that option C would be best. Would you like me to open a ticket?

Until this functionality is baked into a release and as we're not running
1.1 yet (i.e. not using parametrised classes) we may take a stop gap
approach and look toward going back to site.pp as Steve suggested. If we
want to use parametrised classes via Foreman in the future we could drop
empty modules into a common-modules directory as per Greg's suggestion,
while simpler than my original solution the housekeeping aspect puts us off
accepting that approach unless we have to. However, we are probably going
to use a common-modules approach as a location for externally
developed modules taken from puppetforge and github in order to keep our
internal repository clean.

Regards,
Charlie