Foreman and Infrastructure as Code

Support
1

Hi there,

I'm trying to understand the concepts underlying Foreman. In other words,
"how is Foreman meant to be used?"

I've installed Foreman on a VM, played with it for a few hours and browsed
through the documentation. For what Puppet is concerned, it looks like
Foreman expects that you create Puppet classes on the Foreman UI, but you
have to install modules from the command line, directly on the server.
Later, you have to issue a command from to UI to sync the changes in the
file system with the Foreman database.

To me, this seems a bit uncoherent. It's a bit like there is a piece in the
middle not yet covered by the UI (i.e. the installation and/or creation +
modification of modules). It's especially disturbing that you have to
connect with SSH and run commands manually. You could say, this is mainly a
cosmetic issue, a usability concern, but it's really an anti-pattern in
continuous delivery. Servers should be locked down, nobody should be
allowed to make uncontrolled changes. Since none of all the changes are
under version control it's impossible to do rollbacks, and there's probably
not even an audit trail for the changes to the host, neither for the Puppet
configuration nor for changes to Foreman.

The whole situation feels a bit uncomfortable. The Puppet configuration is
an important asset, it's source code that I want to have revisioned,
quality checked (linted), and saved offsite in case the server burns down,
so it can be restored reliably if needed. I could now install Git on the
Foreman host, put /etc/puppetlabs/puppet/code/ under version control, and
deploy changes via a CI server. The problem here is that I have to throw
away any changes (to the Puppet configuration) made by Foreman, otherwise
there will a polluted system, or even a merge conflict.

In other words, Forman is not made with version control in mind, is this
correct? Can anyone confirm this finding?

Can we do something about it? - Think this: If the Foreman host were
actually locked down the only way to introduce changes to the host were by
changing the Puppet configuration that sets up the host itself (which,
fortunately, is already under control of Puppet/Foreman). If the Puppet
configuration on the host could be pushed to a Git repository, and changes
to it be committed and deployed (automatically) back to the host then the
situation would look more promising. The UI to modify the Puppet
configuration would have to be read-only, or it would have to make changes
directly on the main branch of the repository and commit (and push) them
right after any modification.

Peter

2

> I've installed Foreman on a VM, played with it for a few hours and
> browsed through the documentation. For what Puppet is concerned, it
> looks like Foreman expects that you create Puppet classes on the Foreman
> UI, but you have to install modules from the command line, directly on
> the server

You don't "have" to do this, you should probably deploy the Puppet
environment and modules from source control using r10k or
librarian-puppet. Installing modules by hand is fine for an example
(e.g. in documentation), but you should have a better way to do it in
practice.

> Later, you have to issue a command from to UI to sync the
> changes in the file system with the Foreman database.

You can also run foreman-rake puppet:import:puppet_classes[batch]
after deploying the environment change(s).

> Since none of all
> the changes are under version control it's impossible to do rollbacks,
> and there's probably not even an audit trail for the changes to the
> host, neither for the Puppet configuration nor for changes to Foreman.

Changes to hosts and their assigned Puppet classes made through the
Foreman UI should have audit entries created under Monitor > Audits, but
there's no rollback mechanism.

As you say, changes to the modules in the deployed Puppet environments
can, and probably should, be version controlled. This isn't done within
Foreman.

> The whole situation feels a bit uncomfortable. The Puppet configuration
> is an important asset, it's source code that I want to have revisioned,
> quality checked (linted), and saved offsite in case the server burns
> down, so it can be restored reliably if needed. I could now install Git
> on the Foreman host, put /etc/puppetlabs/puppet/code/ under version
> control, and deploy changes via a CI server. The problem here is that I
> have to throw away any changes (to the Puppet configuration) made by
> Foreman, otherwise there will a polluted system, or even a merge conflict.

Foreman doesn't modify Puppet environments, manifests or any other
on-disk data in your Puppet environment, it only imports from them. This
sounds quite reasonable and sensible.

The data in Foreman is mostly associations between its hosts and the
modules and classes in the environment ("classification"). You can also
set top-level class parameters.

> Can we do something about it? - Think this: If the Foreman host were
> actually locked down the only way to introduce changes to the host were
> by changing the Puppet configuration that sets up the host itself
> (which, fortunately, is already under control of Puppet/Foreman). If the
> Puppet configuration on the host could be pushed to a Git repository,
> and changes to it be committed and deployed (automatically) back to the
> host then the situation would look more promising. The UI to modify the
> Puppet configuration would have to be read-only, or it would have to
> make changes directly on the main branch of the repository and commit
> (and push) them right after any modification.

You can manage classification and host-level data in either Foreman (or
another ENC) or within the Puppet environment through node definitions
or Puppet data lookup (Hiera) etc.

If you want to version control host/node classification, then you'd be
much better off doing that inside your Puppet manifests and data files,
then deploying it alongside the modules in your Puppet environment from
source control. Foreman would just be used for reporting and provisioning.

If you want to use Foreman's web UI to manage hosts rather than source
control, then do so, and you can still deploy modules in the Puppet
environment from source control.

··· On 21/05/17 22:21, Peter Bittner wrote:

–
Dominic Cleal
dominic@cleal.org

3

Dominic,

> > looks like Foreman expects that you create Puppet classes on the Foreman
> > UI, but you have to install modules from the command line, directly on
> > the server
>
> You don't "have" to do this, you should probably deploy the Puppet
> environment and modules from source control using r10k or
> librarian-puppet. Installing modules by hand is fine for an example
> (e.g. in documentation), but you should have a better way to do it in
> practice.
>

Thanks for pointing this out. Solves one problem. So, I can add
dependencies in Modulefile or metadata.json, and let our CI server deploy
the changes to the Foreman host.

> Later, you have to issue a command from to UI to sync the
> > changes in the file system with the Foreman database.
>
> You can also run foreman-rake puppet:import:puppet_classes[batch]
> after deploying the environment change(s).
>

Fantastic, thanks for the hint!

Changes to hosts and their assigned Puppet classes made through the
> Foreman UI should have audit entries created under Monitor > Audits, but
> there's no rollback mechanism.
>

A question aside: Where is the Puppet configuration that was (is) used by
foreman-install? I suspected it to live under
/etc/puppetlabs/code/environments/production/, but no it's not. I need
the configuration of the Foreman host be under version control right after
completing the installation. Only this will allow me to apply changes to
the host in a controlled manner. Is it thrown away after the installation
was completed?

> As you say, changes to the modules in the deployed Puppet environments
> can, and probably should, be version controlled. This isn't done within
> Foreman.
>

That's fine as long as I can set this up during the initial setup of the
Foreman host.

Foreman doesn't modify Puppet environments, manifests or any other
> on-disk data in your Puppet environment, it only imports from them. This
> sounds quite reasonable and sensible.
>

Good.

If you want to version control host/node classification, then you'd be
> much better off doing that inside your Puppet manifests and data files,
> then deploying it alongside the modules in your Puppet environment from
> source control. Foreman would just be used for reporting and provisioning.
>

That sounds like what I want.

If you want to use Foreman's web UI to manage hosts rather than source
> control, then do so, and you can still deploy modules in the Puppet
> environment from source control.
>

I don't get that, sorry. What do you intend with "manage host rather than
source control" (through Foreman's UI)?

I want the configuration changes to happen through source control only.
Forman only for transparency, i.e. reporting, sounds perfect.

Peter

4

> Dominic,
>
>
> > looks like Foreman expects that you create Puppet classes on the
> Foreman
> > UI, but you have to install modules from the command line,
> directly on
> > the server
>
> You don't "have" to do this, you should probably deploy the Puppet
> environment and modules from source control using r10k or
> librarian-puppet. Installing modules by hand is fine for an example
> (e.g. in documentation), but you should have a better way to do it in
> practice.
>
>
> Thanks for pointing this out. Solves one problem. So, I can add
> dependencies in |Modulefile| or |metadata.json|, and let our CI server
> deploy the changes to the Foreman host.

Yes, Modulefile usually.

> Changes to hosts and their assigned Puppet classes made through the
> Foreman UI should have audit entries created under Monitor > Audits,
> but
> there's no rollback mechanism.
>
>
> A question aside: Where is the Puppet configuration that was (is) used
> by foreman-install? I suspected it to live under
> /etc/puppetlabs/code/environments/production/, but no it's not. I need
> the configuration of the Foreman host be under version control right
> after completing the installation. Only this will allow me to apply
> changes to the host in a controlled manner. Is it thrown away after the
> installation was completed?

The modules are in /usr/share/foreman-installer/modules, and the
parameters used are in
/etc/foreman-installer/scenarios.d/foreman-answers.yaml. You can add the
modules to your modulepath (via an environment.conf), or fetch them from
Puppet Forge etc.

> If you want to version control host/node classification, then you'd be
> much better off doing that inside your Puppet manifests and data files,
> then deploying it alongside the modules in your Puppet environment from
> source control. Foreman would just be used for reporting and
> provisioning.
>
>
> That sounds like what I want.
>
> If you want to use Foreman's web UI to manage hosts rather than source
> control, then do so, and you can still deploy modules in the Puppet
> environment from source control.
>
>
> I don't get that, sorry. What do you intend with "manage host rather
> than source control" (through Foreman's UI)?

Editing hosts via the web UI (or API) to assign classes, assign groups,
modify class parameters etc.

> I want the configuration changes to happen through source control only.
> Forman only for transparency, i.e. reporting, sounds perfect.

Then I'd suggest putting everything into the Puppet environment, forget
Foreman's Puppet class and parameter handling and use Puppet's data
lookup and Hiera to assign classes and parameters to hosts.

··· On 22/05/17 14:38, Peter Bittner wrote:

–
Dominic Cleal
dominic@cleal.org

5

Dominic,

>
> > I want the configuration changes to happen through source control only.
> > Forman only for transparency, i.e. reporting, sounds perfect.
>
> Then I'd suggest putting everything into the Puppet environment, forget
> Foreman's Puppet class and parameter handling and use Puppet's data
> lookup and Hiera to assign classes and parameters to hosts.
>

That still sounds a bit like a half-baked solution.

  1. Why is the Foreman host being set up and configured using a Puppet
    recipe, and then I should be throwing it away instead of maintaining,
    modifying, extending it?
  2. What is the cleanest path to add the Foreman host to the Puppet
    configuration? (I'm not sure how the foreman-answers.yaml is integrated,
    and whether I should copy everything over from
    /usr/share/foreman-installer/
    modules)
  3. Also, shouldn't the foreman-installer be uninstalled to maintain a
    slim, clean system?

I really don't want to SSH into the Foreman host after setting it up, and
install stuff manually. How do I set things up that a Forman system is set
up, including a Puppet configuration under version control in
/etc/puppetlabs, practically from zero? Every step I need to do manually is
somewhat wasted time, and potential to introduce errors. I want to avoid
that.

Thanks, Peter

··· Am Montag, 22. Mai 2017 16:25:02 UTC+2 schrieb Dominic Cleal:
6

Hi Peter,

>> > I want the configuration changes to happen through source control only.
>> > Forman only for transparency, i.e. reporting, sounds perfect.

>> Then I'd suggest putting everything into the Puppet environment, forget
>> Foreman's Puppet class and parameter handling and use Puppet's data
>> lookup and Hiera to assign classes and parameters to hosts.

> That still sounds a bit like a half-baked solution.

> 1. Why is the Foreman host being set up and configured using a Puppet recipe,
> and then I should be throwing it away instead of maintaining, modifying,
> extending it?
> 2. What is the cleanest path to add the Foreman host to the Puppet
> configuration? (I'm not sure how the foreman-answers.yaml is integrated, and
> whether I should copy everything over from /usr/share/foreman-installer/
> modules )

> 3. Also, shouldn't the foreman-installer be uninstalled to maintain a slim,
> clean system?

> I really don't want to SSH into the Foreman host after setting it up, and
> install stuff manually. How do I set things up that a Forman system is set up,
> including a Puppet configuration under version control in /etc/puppetlabs ,
> practically from zero? Every step I need to do manually is somewhat wasted
> time, and potential to introduce errors. I want to avoid that.

Unless you are in a situation where you are installing new Foreman hosts on a regular basis I think you're being a bit extreme. There is a point at which automation no longer gives you a return on investment.

You may want to look into Nailgun to help automate some of your setup:

Regards,

j

7

> That still sounds a bit like a half-baked solution.
>
> 1. Why is the Foreman host being set up and configured using a Puppet
> recipe, and then I should be throwing it away instead of
> maintaining, modifying, extending it?

Sure, by all means. The modules used in the installer are published for
this reason, so they can be re-used. You don't even need to use the
installer if you use the same set of modules (and the Puppetfile is
published at
https://github.com/theforeman/foreman-installer/tree/1.15-stable). Using
them directly is reasonably advanced, but by all means do.

> 2. What is the cleanest path to add the Foreman host to the Puppet
> configuration? (I'm not sure how the foreman-answers.yaml is
> integrated, and whether I should copy everything over from
> /usr/share/foreman-installer/
> modules)

Probably not to use the installer and to use the modules directly,
either from an existing Puppet master or using puppet apply. Or create
a Puppet environment after installation.

> 3. Also, shouldn't the foreman-installer be uninstalled to maintain a
> slim, clean system?

If you wish, it isn't required at runtime.

··· On 22/05/17 20:55, Peter Bittner wrote:

–
Dominic Cleal
dominic@cleal.org

8

>
> I really don't want to SSH into the Foreman host after setting it up, and
> install stuff manually. How do I set things up that a Forman system is set
> up, including a Puppet configuration under version control in
> /etc/puppetlabs, practically from zero? Every step I need to do manually
> is somewhat wasted time, and potential to introduce errors. I want to avoid
> that.
>
> Unless you are in a situation where you are installing new Foreman hosts
> on a regular basis I think you're being a bit extreme. There is a point at
> which automation no longer gives you a return on investment.
>

No, I'm not extreme. I need consistency and the ability to reproduce an
environment exactly and very fast (e.g. when we need to do trouble shooting
or recovery). Setting up configuration management is only a side-business
in our DevOps-driven business, one puzzle piece of many. We need to unleash
all powers for implementing product features. The less time we spend
managing our infrastructure and doing trouble shooting when things go
wrong, the better.

The behavior I describe should go into the installer - or more generally
into the overall setup procedure -, and everyone should be encouraged to do
it "the right way". No more tinkering on the Foreman host, no more
uncontrolled changes when you use Foreman (and Puppet). When we think in
terms of "open source" we're of course installing new Foreman hosts on a
regular basis. So, we're all set.

Peter

9

> > 2. What is the cleanest path to add the Foreman host to the Puppet
> > configuration? (I'm not sure how the foreman-answers.yaml is
> > integrated, and whether I should copy everything over from
> > /usr/share/foreman-installer/
> > modules)
>
> Probably not to use the installer and to use the modules directly,
> either from an existing Puppet master or using puppet apply. Or create
> a Puppet environment after installation.
>

Can we add instructions for such a scenario to the installation docs?

Installing puppet, installing the modules, and running puppet apply - if
that's all - sounds nice to me. And not too advanced in the sense of "needs
expert knowledge".

Peter

10

> 2. What is the cleanest path to add the Foreman host to the Puppet
>> > configuration? (I'm not sure how the foreman-answers.yaml is
>> > integrated, and whether I should copy everything over from
>> > /usr/share/foreman-installer/
>> > modules)
>>
>> Probably not to use the installer and to use the modules directly,
>> either from an existing Puppet master or using puppet apply. Or create
>> a Puppet environment after installation.
>>
>
> Can we add instructions for such a scenario to the installation docs?
>
> Installing puppet, installing the modules, and running puppet apply - if
> that's all - sounds nice to me. And not too advanced in the sense of "needs
> expert knowledge".
>

Probably, add r10k to the game would make the version-controlled setup more
automatic, wouldn't it? Is there an up-to-date guide or tutorial around
that walks you through setting up Foreman with r10k? I've only found a few
older discussions and posts, e.g.

This one looks like it could be integrated in the docs, maybe in a
streamlined version:

Peter

11

Use r10k's existing documentation, there's really very little overlap
between Foreman and r10k. This is all just configuration of your Puppet
master, not Foreman.

The only way Foreman's involved is that you need to re-run its import
after adding/removing/changing classes using the earlier command or
Foreman's API. You can do this from any script running r10k or with its
postrun hook.

··· On 25/05/17 17:44, Peter Bittner wrote: > > > > 2. What is the cleanest path to add the Foreman host to the > Puppet > > configuration? (I'm not sure how the foreman-answers.yaml is > > integrated, and whether I should copy everything over from > > /usr/share/foreman-installer/ > > modules) > > Probably not to use the installer and to use the modules directly, > either from an existing Puppet master or using `puppet apply`. > Or create > a Puppet environment after installation. > > > Can we add instructions for such a scenario to the installation docs? > > Installing puppet, installing the modules, and running `puppet > apply` - if that's all - sounds nice to me. And not too advanced in > the sense of "needs expert knowledge". > > > Probably, add r10k to the game would make the version-controlled setup > more automatic, wouldn't it? Is there an up-to-date guide or tutorial > around that walks you through setting up Foreman with r10k? I've only > found a few older discussions and posts

–
Dominic Cleal
dominic@cleal.org