Community Template Repository (and Plugin)

Hi all,

At last week's Community Roadmap Meeting, we discussed a central repository
for provisioning templates, and ways to use them more effectively. This
email is to pass on what already exists in tis area, and to discuss how to
make it better.

There is already a github repo containing templates for Fedora/CentOS/RHEL
and Debian/Ubuntu[1].
Currently it contains PXElinux, Kickstart/Preseed, & Finish templates, a
basic Disklayout, and some snippets. These should work for the relevant
OSes, but further contributions are welcome - as are new OS templates :slight_smile:

This is all very well, but you're not going to manually cut & paste those
templates into Foreman, now are you? What is needed is some form
of synchronization. To that end, I wrote a simple plugin called
foreman_templates[2]

This plugin adds a rake task called templates:sync. This will git-clone the
community-templates repo to a tmpdir, and then iterate over it's contents,
creating (or updating) the templates in Foreman. By default, it creates
templates with names like:

Community Fedora(17) PXElinux

If a template exists, the raketask will simply update the content of the
template - existing OS/Environment/Hostgroup assoications are not altered.
If the template is created, it will attempt to figure out which OS/family
it should be associated with.

There are set of options to the rake task, these are documented on the
github page. The most notable two are:

  • repo=<url> - This allow you to use a different git repo - so you can
    store your own templates in a git repo now :wink:
  • filter=<regex> - This allows you to import only a subset of templates.
    For example, I tested the Fedora ones using filter="fedora"

There's more that can be done with this plugin (I'd like to add a button to
the UI, for a start :P) but hopefully some of you will find it useful.

Cheers,
Greg

[1]https://github.com/theforeman/community-templates
[2] https://github.com/GregSutcliffe/foreman_templates

Great stuff.

Does the rake task insert audit (so I am able to revert back when
something wrong happens)?

I know the history will be in git, but I mean for the initial
"overwrite"…

LZ

··· On Mon, May 06, 2013 at 02:02:46PM +0100, Greg Sutcliffe wrote: > Hi all, > > At last week's Community Roadmap Meeting, we discussed a central repository > for provisioning templates, and ways to use them more effectively. This > email is to pass on what already exists in tis area, and to discuss how to > make it better. > > There is already a github repo containing templates for Fedora/CentOS/RHEL > and Debian/Ubuntu[1]. > Currently it contains PXElinux, Kickstart/Preseed, & Finish templates, a > basic Disklayout, and some snippets. These should work for the relevant > OSes, but further contributions are welcome - as are new OS templates :) > > This is all very well, but you're not going to manually cut & paste those > templates into Foreman, now are you? What is needed is some form > of synchronization. To that end, I wrote a simple plugin called > foreman_templates[2] > > This plugin adds a rake task called templates:sync. This will git-clone the > community-templates repo to a tmpdir, and then iterate over it's contents, > creating (or updating) the templates in Foreman. By default, it creates > templates with names like: > > Community Fedora(17) PXElinux > > If a template exists, the raketask will simply update the content of the > template - existing OS/Environment/Hostgroup assoications are not altered. > If the template is created, it will attempt to figure out which OS/family > it should be associated with. > > There are set of options to the rake task, these are documented on the > github page. The most notable two are: > > * repo= - This allow you to use a different git repo - so you can > store your own templates in a git repo now ;) > * filter= - This allows you to import only a subset of templates. > For example, I tested the Fedora ones using filter="fedora" > > There's more that can be done with this plugin (I'd like to add a button to > the UI, for a start :P) but hopefully some of you will find it useful. > > Cheers, > Greg > > [1]https://github.com/theforeman/community-templates > [2] https://github.com/GregSutcliffe/foreman_templates > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > >


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

> Hi all,
>
> At last week's Community Roadmap Meeting, we discussed a central
> repository for provisioning templates, and ways to use them more
> effectively. This email is to pass on what already exists in tis area, and
> to discuss how to make it better.
>
> There is already a github repo containing templates for Fedora/CentOS/RHEL
> and Debian/Ubuntu[1].
> Currently it contains PXElinux, Kickstart/Preseed, & Finish templates, a
> basic Disklayout, and some snippets. These should work for the relevant
> OSes, but further contributions are welcome - as are new OS templates :slight_smile:
>
> This is all very well, but you're not going to manually cut & paste those
> templates into Foreman, now are you? What is needed is some form
> of synchronization. To that end, I wrote a simple plugin called
> foreman_templates[2]
>
> This plugin adds a rake task called templates:sync. This will git-clone
> the community-templates repo to a tmpdir, and then iterate over it's
> contents, creating (or updating) the templates in Foreman. By default, it
> creates templates with names like:
>
> Community Fedora(17) PXElinux
>
> If a template exists, the raketask will simply update the content of the
> template - existing OS/Environment/Hostgroup assoications are not altered.
> If the template is created, it will attempt to figure out which OS/family
> it should be associated with.
>
> There are set of options to the rake task, these are documented on the
> github page. The most notable two are:
>
> * repo=<url> - This allow you to use a different git repo - so you can
> store your own templates in a git repo now :wink:
> * filter=<regex> - This allows you to import only a subset of templates.
> For example, I tested the Fedora ones using filter="fedora"
>
> There's more that can be done with this plugin (I'd like to add a button
> to the UI, for a start :P) but hopefully some of you will find it useful.
>
> Cheers,
> Greg
>
> [1]https://github.com/theforeman/community-templates
> [2] https://github.com/GregSutcliffe/foreman_templates
>

how much work would it be to browser the images and select which one to
copy over? via the UI ? :slight_smile:

also, I don't think you need to add the type of each template to the new
template name.

Ohad

··· On Mon, May 6, 2013 at 4:02 PM, Greg Sutcliffe wrote:


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yes, you do get Audits - which is quite handy if you set this up as a
regular cron job :slight_smile:

Greg

··· On 6 May 2013 14:34, Lukas Zapletal wrote:

Great stuff.

Does the rake task insert audit (so I am able to revert back when
something wrong happens)?

I know the history will be in git, but I mean for the initial
"overwrite"…

>
> how much work would it be to browser the images and select which one to
> copy over? via the UI ? :slight_smile:
>

Some kind of UI wizard did occur to me - much like the Locs/Orgs, we could
display all the proposed templates, and click/drag the ones we want over
the the list to be applied. I've no idea how much work that is though :wink:

> also, I don't think you need to add the type of each template to the new
> template name.
>

Possibly not - I didn't want 3 templates all called "Community Fedora 17"
though, so I used the template type to make them unique. I'm happy to take
suggestions on other ways to achieve this :slight_smile:

Side issues that might be worth addressing in Foreman core - A mass-delete
function (like we can do with hosts?) and a clone function would both be
useful :slight_smile:

··· On 6 May 2013 15:08, Ohad Levy wrote:

This completely snuck under my radar, I must have been on holiday. What an
awesome feature! I've always had this underlying worry that while all my
puppet manifests are under solid version control with traceability back to
the issue tracker my kickstart scripts weren't (and they used to be),
especially as they do some real heavy lifting with complex disk
partitioning etc

I'll give this a go ASAP and get some feedback back to you Greg.

Charlie

··· On Mon, May 6, 2013 at 4:35 PM, Greg Sutcliffe wrote:

On 6 May 2013 14:34, Lukas Zapletal lzap@redhat.com wrote:

Great stuff.

Does the rake task insert audit (so I am able to revert back when
something wrong happens)?

I know the history will be in git, but I mean for the initial
"overwrite"…

Yes, you do get Audits - which is quite handy if you set this up as a
regular cron job :slight_smile:

Greg


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

> This completely snuck under my radar, I must have been on holiday. What an
> awesome feature! I've always had this underlying worry that while all my
> puppet manifests are under solid version control with traceability back to
> the issue tracker my kickstart scripts weren't (and they used to be),
> especially as they do some real heavy lifting with complex disk
> partitioning etc
>

Pretty much why I wrote it - it solves both a collaboration problem and a
versioning problem. I'd recommend doing everything in git and then running
the rake task regularly from cron (after suitable testing ofc :P)

> I'll give this a go ASAP and get some feedback back to you Greg.
>

Fantastic! It's fallen down my todo list a bit, but I'd still like to get
some more feature improvements into it. The new template-metadata support
in the plugin helps with automatic OS association though :slight_smile:

Looking forward to finding out your thoughts.
Greg

··· On 28 September 2013 09:31, Charlie Derwent wrote:

>
>> This completely snuck under my radar, I must have been on holiday. What
>> an awesome feature! I've always had this underlying worry that while all my
>> puppet manifests are under solid version control with traceability back to
>> the issue tracker my kickstart scripts weren't (and they used to be),
>> especially as they do some real heavy lifting with complex disk
>> partitioning etc
>>
>
> Pretty much why I wrote it - it solves both a collaboration problem and a
> versioning problem. I'd recommend doing everything in git and then running
> the rake task regularly from cron (after suitable testing ofc :P)
>
>
>> I'll give this a go ASAP and get some feedback back to you Greg.
>>
>
> Fantastic! It's fallen down my todo list a bit, but I'd still like to get
> some more feature improvements into it. The new template-metadata support
> in the plugin helps with automatic OS association though :slight_smile:
>
> Looking forward to finding out your thoughts.
>

Hi Greg,

Sorry it's taken so long to have a look at your plugin, firstly it works
and was ridiculously easy to get going (which is always awesome!), I
ignored the message about the "unknown template type 'iPXE'" as I saw Dom's
Pull Request comments. As the plugin is already so useful in its current
form we will start adapting our processes to run it in production (gulp).

The one thing that is missing, which I'd argue is even more important than
the button in UI, is rather than using the cron idea you suggested above,
is using the rake task to sync tags as well as specific branches via git
hooks on commits. I could add the basic functionality myself and send it
back to you in a PR pretty quickly, but I realised it probably needs a bit
of thought around error handling to stop someone trying to sync a branch
and a tag in the same command. Unfortunately I'm the worst ruby coder in
the world and for now I am comfortable just letting it sync to the master
branch (which in our setup tracks the latest tag).

The elephant in the room though is whether or not provisioning scripts
should follow the same "dev > test > production" environment workflow that
the puppet modules go through; possibly even living within the same
repository as the puppet modules(using your dirname option would enable
that, but running a full-blown git clone on every update promotes keeping
the provisioning repo as small as possible). Aligning the
workflows/repositories feels like the right thing to do to, in effect,
create a genuinely complete infrastructure code repository with foreman
managing the data and pulling the strings.

The main blocker is Foreman isn't really designed to handle provisioning in
this way. While some files can be explicitly tied to an environment they
all need unique names and snippets currently have no concept of an
environment at all.

I've managed to get round this by using your prefix option to denote the
environment that the scripts and snippets belong to then using things like
<%= snippet "#{@host.environment} puppet.conf" %> in my provisioning
templates to dynamically load the right one, but it does feel a little bit
hacky.

With your plugin the only thing that I think you could do to help is to
add automatic environment association as a template:sync option (rather
than in the metadata). Either that or the core product could port some of
the "checkbox -> select action -> change environment" goodness from the
hosts page onto the provisioning templates page.

I also think it would be worthwhile for someone to think about how you can
support the interactions with SCM in a less hands-on way for end users,
almost as if you could point foreman at a remote repo and it will parse
your branches and tags to particular defined environments (for puppet
modules, provisioning templates or anything else). With all the talk in the
dev channel about using Git to drive Katello it feels like everybody is
moving in the same direction.

tl;dr - Awesome plugin - Could you add tag support and
automatic environment association in the template:sync command? Foreman
core needs some work.

Cheers,
Charlie

··· On Sat, Sep 28, 2013 at 12:39 PM, Greg Sutcliffe wrote: > On 28 September 2013 09:31, Charlie Derwent wrote:

Greg


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.

For more options, visit https://groups.google.com/groups/opt_out.

> Sorry it's taken so long to have a look at your plugin, firstly it works
> and was ridiculously easy to get going (which is always awesome!), I
> ignored the message about the "unknown template type 'iPXE'" as I saw Dom's
> Pull Request comments. As the plugin is already so useful in its current
> form we will start adapting our processes to run it in production (gulp).
>

Glad to hear it - your timing is perfect as the latest version was only
released last week :slight_smile:

> The one thing that is missing, which I'd argue is even more important than
> the button in UI, is rather than using the cron idea you suggested above,
> is using the rake task to sync tags as well as specific branches via git
> hooks on commits. I could add the basic functionality myself and send it
> back to you in a PR pretty quickly, but I realised it probably needs a bit
> of thought around error handling to stop someone trying to sync a branch
> and a tag in the same command. Unfortunately I'm the worst ruby coder in
> the world and for now I am comfortable just letting it sync to the master
> branch (which in our setup tracks the latest tag).
>

To the best of my knowledge, it should work fine with a tag - the code just
calls "git clone -b $branch" but $branch works equally well as a tag. Git
doesn't differentiate between a tag or a branch (at least, not since quite
an old version number). For example:

[greg:/tmp]$ git clone
https://github.com/GregSutcliffe/community-templates-b this_tag
Cloning into 'community-templates'…
remote: Counting objects: 380, done.
remote: Compressing objects: 100% (253/253), done.
remote: Total 380 (delta 142), reused 302 (delta 88)
Receiving objects: 100% (380/380), 50.28 KiB | 0 bytes/s, done.
Resolving deltas: 100% (142/142), done.
Checking connectivity… done.
Note: checking out 'f2debb0691b2bc8ae88c4897835139743c2ad341'.

Give it a try and see if it works as is?

The elephant in the room though is whether or not provisioning scripts
> should follow the same "dev > test > production" environment workflow that
> the puppet modules go through; possibly even living within the same
> repository as the puppet modules(using your dirname option would enable
> that, but running a full-blown git clone on every update promotes keeping
> the provisioning repo as small as possible). Aligning the
> workflows/repositories feels like the right thing to do to, in effect,
> create a genuinely complete infrastructure code repository with foreman
> managing the data and pulling the strings.
>
> The main blocker is Foreman isn't really designed to handle provisioning
> in this way. While some files can be explicitly tied to an environment they
> all need unique names and snippets currently have no concept of an
> environment at all.
>
> I've managed to get round this by using your prefix option to denote the
> environment that the scripts and snippets belong to then using things like
> <%= snippet "#{@host.environment} puppet.conf" %> in my provisioning
> templates to dynamically load the right one, but it does feel a little bit
> hacky.
>

This is great feedback - do open issues at [1] where you feel we can make
this better.

With your plugin the only thing that I think you could do to help is to
> add automatic environment association as a template:sync option (rather
> than in the metadata). Either that or the core product could port some of
> the "checkbox -> select action -> change environment" goodness from the
> hosts page onto the provisioning templates page.
>

I think both are valid - do raise feature requests for these. I think it
would be pretty trivial to have the plugin override the metadata when it's
specified on the cmdline.

> I also think it would be worthwhile for someone to think about how you can
> support the interactions with SCM in a less hands-on way for end users,
> almost as if you could point foreman at a remote repo and it will parse
> your branches and tags to particular defined environments (for puppet
> modules, provisioning templates or anything else). With all the talk in the
> dev channel about using Git to drive Katello it feels like everybody is
> moving in the same direction.
>

So if I'm reading you aright, if we detect a branch structure like
'environments/foo' then we would automatically assign all the templates in
that branch to env 'foo'? That's a really interesting idea… feature
request at [2] please :slight_smile:

tl;dr - Awesome plugin - Could you add tag support and
> automatic environment association in the template:sync command? Foreman
> core needs some work.
>

Yep, just raise the issues :slight_smile:

Thanks for the feedback!
Greg

[1] Foreman
[2] Foreman

··· On 9 December 2013 04:25, Charlie Derwent wrote: