New installer and puppet modules

Hi all,

I'm working on new puppet-based installer that will be able to install foreman
and katello. Two questions were raised recently.

#1 Do we want installer repository to contain puppet modules we need?
#2 If yes, how do we want to link them in git repository? (submodules,
subtree, …)

#1
We could distribute modules in different way (e.g. install them from
puppetforge). However this would require us to put all our modules to there.
Also we could lose some ability to control them? Especially if we had to
extend some functionality of existing module which is not under our control.
Maybe someone with deep knowledge of puppet could tell whether there is
something like bundler for puppet modules (chef has a tool called librarian
for this).

My preference for #1 would be to have puppet modules as a part of installer
repo.

#2
There are known problems using submodules. Namely:

  • deleting them is not easy task,
  • switching branches where one time it's directory, second time it's a
    submodule

I'd say both can happen quite easily in our use case.

Using git subtrees [1] also have some limitations. It's not easy to push
changes back to upstream when you make changes to a subtree. However I don't
think we need this. Also we get ability to make a patch to a puppet module
that we need before it get's to upstream (again, think about modules we have
from puppetlabs).

My preference for #2 would be using subtrees.

Do you have any other ideas how to achieve this?

[1] http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

··· -- Marek

> Hi all,
>
> I'm working on new puppet-based installer that will be able to install foreman
> and katello. Two questions were raised recently.
>
> #1 Do we want installer repository to contain puppet modules we need?
> #2 If yes, how do we want to link them in git repository? (submodules,
> subtree, …)
>
> #1
> We could distribute modules in different way (e.g. install them from
> puppetforge). However this would require us to put all our modules to there.
> Also we could lose some ability to control them? Especially if we had to
> extend some functionality of existing module which is not under our control.
> Maybe someone with deep knowledge of puppet could tell whether there is
> something like bundler for puppet modules (chef has a tool called librarian
> for this).
>
> My preference for #1 would be to have puppet modules as a part of installer
> repo.

I agree… perhaps publish to puppet forge at releases?

>
> #2
> There are known problems using submodules. Namely:
> * deleting them is not easy task,
> * switching branches where one time it's directory, second time it's a
> submodule
>
> I'd say both can happen quite easily in our use case.
>
> Using git subtrees [1] also have some limitations. It's not easy to push
> changes back to upstream when you make changes to a subtree. However I don't
> think we need this. Also we get ability to make a patch to a puppet module
> that we need before it get's to upstream (again, think about modules we have
> from puppetlabs).
>
> My preference for #2 would be using subtrees.
>

No Opinion.

  • bk
··· On 07/17/2013 07:24 AM, Marek Hulan wrote:

> Hi all,
>
> I'm working on new puppet-based installer that will be able to install foreman
> and katello. Two questions were raised recently.
>
> #1 Do we want installer repository to contain puppet modules we need?
> #2 If yes, how do we want to link them in git repository? (submodules,
> subtree, …)

I'd viewed the work to create a reusable installer library as a separate
thing from the foreman-installer itself.

The main installer repo today is here, which contains a series of
submodules for each individual puppet module:

The submodule repos only contain a master branch (usually).

The installer repo contains the generate_answers.rb script (which I see
being replaced with bin/foreman-installer or similar), the
foreman_installer module (which instantiates classes from YAML input),
the spec file and submodules to everything else. This is the repo that
gets branched on release (develop, stable, master).

I don't see a reason to change this layout, at least, not as part of
this work. This installer repo would remain the finished product,
bringing together the individual modules and the installer script and
would be the one that's released.

I see the kafo as a library that we use, which is released and packaged
separately, then consumed from within bin/foreman-installer and
katello-configure, with the modules that those installers wish to use
being stored in the foreman/katello installer projects (using whatever
impl makes sense or is used today).

> #1
> We could distribute modules in different way (e.g. install them from
> puppetforge). However this would require us to put all our modules to there.
> Also we could lose some ability to control them? Especially if we had to
> extend some functionality of existing module which is not under our control.
> Maybe someone with deep knowledge of puppet could tell whether there is
> something like bundler for puppet modules (chef has a tool called librarian
> for this).
>
> My preference for #1 would be to have puppet modules as a part of installer
> repo.

We do now release our modules to Puppet Forge:

You could use these with librarian-puppet, but I don't really see where
this fits in with the installer case. We would want to continue
shipping the installer as a package, complete with all the modules it
needs and not to pull them down from an external resource.

A number of people use these modules directly, so that must continue to
work. If they want to use puppet module install, pulp, r10k, or
librarian-puppet to do so, they can. I don't think that affects how our
installer should work though.

> #2
> There are known problems using submodules. Namely:
> * deleting them is not easy task,
> * switching branches where one time it's directory, second time it's a
> submodule
>
> I'd say both can happen quite easily in our use case.
>
> Using git subtrees [1] also have some limitations. It's not easy to push
> changes back to upstream when you make changes to a subtree. However I don't
> think we need this. Also we get ability to make a patch to a puppet module
> that we need before it get's to upstream (again, think about modules we have
> from puppetlabs).
>
> My preference for #2 would be using subtrees.
>
> Do you have any other ideas how to achieve this?
>
> [1] http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

The patching is an issue we've hit a few times, and the only way we have
to solve it is to change the remote in .gitmodules to our own fork if
upstream has yet to be merged and it's critical for us. Usually we can
just wait it out.

I'm not familiar with subtrees, so can't comment on one versus the
other, but I'm willing to change this if it's beneficial and others
agree. But I don't think this affects kafo :slight_smile:

··· On 17/07/13 12:24, Marek Hulan wrote:


Dominic Cleal
Red Hat Engineering

librarian-puppet:
https://github.com/rodjek/librarian-puppet

Seems to be quite awesome.

In my puppet repo I have a Gemfile with rspec/librarian/puppet/etc and
then I have a Puppetfile with the modules I need. I only just started
using it though, so I can't be certain exactly how awesome it is, but it
seems pretty awesome so far.

-Jeremy

··· On Wed, Jul 17, 2013 at 01:24:42PM +0200, Marek Hulan wrote: > Hi all, > > I'm working on new puppet-based installer that will be able to install foreman > and katello. Two questions were raised recently. > > #1 Do we want installer repository to contain puppet modules we need? > #2 If yes, how do we want to link them in git repository? (submodules, > subtree, ...) > > #1 > We could distribute modules in different way (e.g. install them from > puppetforge). However this would require us to put all our modules to there. > Also we could lose some ability to control them? Especially if we had to > extend some functionality of existing module which is not under our control. > Maybe someone with deep knowledge of puppet could tell whether there is > something like bundler for puppet modules (chef has a tool called librarian > for this).

Pretty much +1 to everything Dominic said, especially rewriting
generate_answers.rb :wink:

> > My preference for #2 would be using subtrees.
> >
> > Do you have any other ideas how to achieve this?
> >
> > [1]
> http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/
>
> The patching is an issue we've hit a few times, and the only way we have
> to solve it is to change the remote in .gitmodules to our own fork if
> upstream has yet to be merged and it's critical for us. Usually we can
> just wait it out.
>
> I'm not familiar with subtrees, so can't comment on one versus the
> other, but I'm willing to change this if it's beneficial and others
> agree. But I don't think this affects kafo :slight_smile:
>

I have used subtrees in $lastjob, and to be honest, I found then harder to
work with than submodules. It's very easy to make a mistake with where you
commit stuff. Additionally, subtrees are really designed to disguise the
presence of third-party code from the end-user - it just looks like one
repo to them. In our case, where we have independant functioning modules
which can be mixed and matched, I think it plays to our strength to display
those modules clearly as submodules of the installer. In short, -1 to
subtrees :wink:

Greg

··· On 17 July 2013 13:42, Dominic Cleal wrote:

> Pretty much +1 to everything Dominic said, especially rewriting
> generate_answers.rb :wink:

I am also for git submodules, franky I still hate them, but for
foreman-installer it is kinda working fine.

Just one note: I work directly in submodules after advice from Dom and
it works just fine. Since the master git repo act more as a "container"
of submodules, it does not bring any big issues. There are some files,
but not many.

Maybe this is great opportunity to push this a little bit forward - I
vote for using main git repo solely for submodules with the following
structure (something like):

./.gitmodules
./cafo-configure (submodule)
./manifests/apache2 (submodule)
./manifests/qpid (submodule)
./manifests/katello (submodule)

Keeping the main git repo clean makes our life much easier:

cd manifests/apache2

git remote add lzap …

hack

Only when we consider that modules are to be released for public, we can
update .gitmodules file via normal git module subcommands.

LZ

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman