Packaging of plugins

I'm working on a story to package the discovery plugin[1] as an RPM, but
want to discuss how we distribute plugins more generally.

While I think packaging a few common plugins as RPMs is useful for
people who either a) prefer OS packages instead of gems, b) don't have
direct internet access, we should still work on better OS independent
support for plugins. I'd quite like something similar to Jenkins' level
of support for installing and maintaining them via a vendored installation.

However for RPMs specifically, I'd propose the following:

  1. we encourage plugin authors to move stable and/or popular plugins
    into the 'theforeman' GitHub organisation. Discovery has already done this.

1a) we add a GitHub team for them to retain commit and administrative
access. This is similar to Jenkins[2], and it allows another volunteer
to easily take over maintenance if the author moves on.

1b) plugins must accept a standard "release" script in the repo that can
do combined gem + RPM releases into koji, similar to the
foreman-katello-engine script[3].

  1. plugins have separate release schedules to Foreman, entirely up to
    the maintainer

  2. we create a new /plugins repo on yum.tf.org with the following layout:

/plugins/1.1 plugins compatible with Foreman 1.1
/plugins/1.2 etc.
/plugins/1.3 if it's compatible with nightlies and eventual 1.3

We create a corresponding set of tags in Koji and set up continuous
publishing into the above repos:

foreman-plugins-1.1-{rhel6,fedora18}
foreman-plugins-1.2-{rhel6,fedora18}
foreman-plugins-1.3-{rhel6,fedora18} etc.

  1. the maintainer decides which tag(s) it's appropriate to release into
    (using our provided script), based on the current compatibility of the
    plugin in relation to Foreman. We can improve this later with metadata
    in the plugin too.

  2. users are then able to configure the plugin repo for their current
    release (and we could do this in the installer) and can install and
    update plugins as and when they wish to.

Thoughts?

[1] https://github.com/theforeman/foreman_discovery
[2] https://wiki.jenkins-ci.org/display/JENKINS/Hosting+Plugins
[3]
https://github.com/Katello/foreman-katello-engine/blob/master/rel-eng/release.sh

··· -- Dominic Cleal Red Hat Engineering

I forgot to mention that we'd also help by building any dependencies
necessary for plugins into these tags and therefore into the /plugins
repos. I don't think this would be many.

If we built dependencies for plugins into the core Foreman repos, it'd
be difficult to change after each Foreman release. One of my aims is to
ensure plugin developers can move at a different pace to that of the
core Foreman release cycle.

··· On 28/06/13 10:54, Dominic Cleal wrote: > 3) we create a new /plugins repo on yum.tf.org with the following layout: > > /plugins/1.1 plugins compatible with Foreman 1.1 > /plugins/1.2 etc. > /plugins/1.3 if it's compatible with nightlies and eventual 1.3 > > We create a corresponding set of tags in Koji and set up continuous > publishing into the above repos: > > foreman-plugins-1.1-{rhel6,fedora18} > foreman-plugins-1.2-{rhel6,fedora18} > foreman-plugins-1.3-{rhel6,fedora18} etc.


Dominic Cleal
Red Hat Engineering

I apologize for bringing an old thread to life.

> 1) we encourage plugin authors to move stable and/or popular plugins
> into the 'theforeman' GitHub organisation. Discovery has already done this.
>
> 1a) we add a GitHub team for them to retain commit and administrative
> access. This is similar to Jenkins[2], and it allows another volunteer
> to easily take over maintenance if the author moves on.
>
> 1b) plugins must accept a standard "release" script in the repo that can
> do combined gem + RPM releases into koji, similar to the
> foreman-katello-engine script[3].

I am more for maintaining SPEC files in our standard thirdparty repo.
Why should be plugins different than other dependencies we already have?

> We create a corresponding set of tags in Koji and set up continuous
> publishing into the above repos:
>
> foreman-plugins-1.1-{rhel6,fedora18}
> foreman-plugins-1.2-{rhel6,fedora18}
> foreman-plugins-1.3-{rhel6,fedora18} etc.

Is this something you are working on? ^

And how about plugins for nightly?

> [1] https://github.com/theforeman/foreman_discovery

I started working on foreman_discovery RPM package, my plan is to commit
it into our thirdparty repo.

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Broadly, looks good.

Packaging from a Debian standpoint isn't too hard (main issue will be
correctly dropping new cache gems into vendor/cache, but this is my problem
;P), but I'm unsure where we should store the debian/ subdir which holds
the packaging. You don't say where the Spec would live for the RPM either,
so I'm no wiser on this.

Greg

> From: "Lukas Zapletal" <lzap@redhat.com>
> To: foreman-dev@googlegroups.com
> Sent: Tuesday, July 30, 2013 9:43:44 AM
> Subject: Re: [foreman-dev] Packaging of plugins
>
> I apologize for bringing an old thread to life.
>
> > 1) we encourage plugin authors to move stable and/or popular plugins
> > into the 'theforeman' GitHub organisation. Discovery has already done
> > this.
> >
> > 1a) we add a GitHub team for them to retain commit and administrative
> > access. This is similar to Jenkins[2], and it allows another volunteer
> > to easily take over maintenance if the author moves on.
> >
> > 1b) plugins must accept a standard "release" script in the repo that can
> > do combined gem + RPM releases into koji, similar to the
> > foreman-katello-engine script[3].
>
> I am more for maintaining SPEC files in our standard thirdparty repo.
> Why should be plugins different than other dependencies we already have?
>
> > We create a corresponding set of tags in Koji and set up continuous
> > publishing into the above repos:
> >
> > foreman-plugins-1.1-{rhel6,fedora18}
> > foreman-plugins-1.2-{rhel6,fedora18}
> > foreman-plugins-1.3-{rhel6,fedora18} etc.
>
> Is this something you are working on? ^

Jan will need to create those tags if they don't already exist.

>
> And how about plugins for nightly?
>
> > [1] https://github.com/theforeman/foreman_discovery
>
> I started working on foreman_discovery RPM package, my plan is to commit
> it into our thirdparty repo.

Why not just put the packaging files into foreman-packaging?

··· ----- Original Message -----


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

> From: "Dominic Cleal" <dcleal@redhat.com>
> To: foreman-dev@googlegroups.com
> Sent: Tuesday, July 30, 2013 10:27:28 AM
> Subject: Re: [foreman-dev] Packaging of plugins
>
> > I apologize for bringing an old thread to life.
> >
> >> 1) we encourage plugin authors to move stable and/or popular plugins
> >> into the 'theforeman' GitHub organisation. Discovery has already done
> >> this.
> >>
> >> 1a) we add a GitHub team for them to retain commit and administrative
> >> access. This is similar to Jenkins[2], and it allows another volunteer
> >> to easily take over maintenance if the author moves on.
> >>
> >> 1b) plugins must accept a standard "release" script in the repo that can
> >> do combined gem + RPM releases into koji, similar to the
> >> foreman-katello-engine script[3].
> >
> > I am more for maintaining SPEC files in our standard thirdparty repo.
> > Why should be plugins different than other dependencies we already have?
> >
> >> We create a corresponding set of tags in Koji and set up continuous
> >> publishing into the above repos:
> >>
> >> foreman-plugins-1.1-{rhel6,fedora18}
> >> foreman-plugins-1.2-{rhel6,fedora18}
> >> foreman-plugins-1.3-{rhel6,fedora18} etc.
> >
> > Is this something you are working on? ^
>
> This is done, we have tags and build targets created, plus comps files
> in foreman-packaging for the plugin repos.
>
> I haven't checked that the repos are generated yet, but they should be
> done on the same schedule as the nightly tags. We'd then need to sync
> them over to yum.tf.org.
>
> > And how about plugins for nightly?
>
> They'd be built for the next version, e.g. 1.3 at the moment.
>
> >> [1] https://github.com/theforeman/foreman_discovery
> >
> > I started working on foreman_discovery RPM package, my plan is to commit
> > it into our thirdparty repo.
>
> I'd started on this, see https://gist.github.com/domcleal/6113370
>
> I would prefer that spec files for projects are kept somewhere a bit
> more accessible really, I think they should live closer to the plugin repo.

+1.

··· ----- Original Message ----- > On 30/07/13 14:43, Lukas Zapletal wrote:


Dominic Cleal
Red Hat Engineering


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

> I apologize for bringing an old thread to life.
>
>> 1) we encourage plugin authors to move stable and/or popular plugins
>> into the 'theforeman' GitHub organisation. Discovery has already done this.
>>
>> 1a) we add a GitHub team for them to retain commit and administrative
>> access. This is similar to Jenkins[2], and it allows another volunteer
>> to easily take over maintenance if the author moves on.
>>
>> 1b) plugins must accept a standard "release" script in the repo that can
>> do combined gem + RPM releases into koji, similar to the
>> foreman-katello-engine script[3].
>
> I am more for maintaining SPEC files in our standard thirdparty repo.
> Why should be plugins different than other dependencies we already have?
>
>> We create a corresponding set of tags in Koji and set up continuous
>> publishing into the above repos:
>>
>> foreman-plugins-1.1-{rhel6,fedora18}
>> foreman-plugins-1.2-{rhel6,fedora18}
>> foreman-plugins-1.3-{rhel6,fedora18} etc.
>
> Is this something you are working on? ^

This is done, we have tags and build targets created, plus comps files
in foreman-packaging for the plugin repos.

I haven't checked that the repos are generated yet, but they should be
done on the same schedule as the nightly tags. We'd then need to sync
them over to yum.tf.org.

> And how about plugins for nightly?

They'd be built for the next version, e.g. 1.3 at the moment.

>> [1] https://github.com/theforeman/foreman_discovery
>
> I started working on foreman_discovery RPM package, my plan is to commit
> it into our thirdparty repo.

I'd started on this, see https://gist.github.com/domcleal/6113370

I would prefer that spec files for projects are kept somewhere a bit
more accessible really, I think they should live closer to the plugin repo.

··· On 30/07/13 14:43, Lukas Zapletal wrote:


Dominic Cleal
Red Hat Engineering

> Jan will need to create those tags if they don't already exist.

I will take care of that. Also will include nigtly tags.

> Why not just put the packaging files into foreman-packaging?

They will end up there - there is a script and task that pull these
automatically from koji…

LZ

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

> I'd started on this, see https://gist.github.com/domcleal/6113370

So can I take this and carry on?

> I would prefer that spec files for projects are kept somewhere a bit
> more accessible really, I think they should live closer to the plugin repo.

Hmm why not. What are your proposals then? But I'd like to put tito
machinery there.

··· On Tue, Jul 30, 2013 at 03:27:28PM +0100, Dominic Cleal wrote:


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

The spec would have to live in the plugin repo.

··· On 28/06/13 11:06, Greg Sutcliffe wrote: > Broadly, looks good. > > Packaging from a Debian standpoint isn't too hard (main issue will be > correctly dropping new cache gems into vendor/cache, but this is my > problem ;P), but I'm unsure where we should store the debian/ subdir > which holds the packaging. You don't say where the Spec would live for > the RPM either, so I'm no wiser on this.


Dominic Cleal
Red Hat Engineering

> From: "Lukas Zapletal" <lzap@redhat.com>
> To: foreman-dev@googlegroups.com
> Sent: Tuesday, July 30, 2013 10:18:39 AM
> Subject: Re: [foreman-dev] Packaging of plugins
>
> > Jan will need to create those tags if they don't already exist.
>
> I will take care of that. Also will include nigtly tags.
>
> > Why not just put the packaging files into foreman-packaging?
>
> They will end up there - there is a script and task that pull these
> automatically from koji…

I know, but why do they need to be in thirdparty at all?

··· ----- Original Message -----

https://github.com/theforeman/foreman-packaging/commits/master

LZ


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

> From: "Lukas Zapletal" <lzap@redhat.com>
> To: foreman-dev@googlegroups.com
> Sent: Tuesday, July 30, 2013 10:36:13 AM
> Subject: Re: [foreman-dev] Packaging of plugins
>
> > I'd started on this, see https://gist.github.com/domcleal/6113370
>
> So can I take this and carry on?
>
> > I would prefer that spec files for projects are kept somewhere a bit
> > more accessible really, I think they should live closer to the plugin repo.
>
> Hmm why not. What are your proposals then? But I'd like to put tito
> machinery there.

I like the idea of having a separate repo for plugin packaging that we can give plugin authors access to in order for them to build packages for their own plugins. If we don't do that then the regular packaging repo in theforeman org on github is fine.

··· ----- Original Message ----- > On Tue, Jul 30, 2013 at 03:27:28PM +0100, Dominic Cleal wrote:


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

>> I'd started on this, see https://gist.github.com/domcleal/6113370
>
> So can I take this and carry on?

Yep.

>> I would prefer that spec files for projects are kept somewhere a bit
>> more accessible really, I think they should live closer to the plugin repo.
>
> Hmm why not. What are your proposals then? But I'd like to put tito
> machinery there.

That's unacceptable, plugin authors may not be able to use tito.

I'd suggest either keeping the spec file within the plugin repo itself,
or a repo like thirdparty within the 'theforeman' organisation that all
plugin maintainers have commit access to (though that's not perfect).
Builds could be triggered through Jenkins to avoid the need for tito,
koji etc for plugin maintainers?

··· On 30/07/13 15:36, Lukas Zapletal wrote: > On Tue, Jul 30, 2013 at 03:27:28PM +0100, Dominic Cleal wrote:


Dominic Cleal
Red Hat Engineering

I am afraid plugin authors are not interested in packaging. These are
two different things. We should keep in mind that packagers (that's
mostly us) should be in our major focus. We should make this easy for
packagers, not plugin authors.

We must keep building in Koji. This is not about a gem or two, all these
plugins will have dependencies and we need to package them, put them in
repos and distribute them. We cannot make this simple process and avoid
tito or koji at all.

Here are my preferences:

A) Let plugin authors to do rubygems releasing, doing our packaging work
using normal means - thirdparty repo, tito, koji, standard processes.

B) Do the same, but creating new github repository, still using tito,
koji, standard process. For each distribution type to create one
subdirectory (debian, redhat) and making these two separate sandboxes.

··· On Tue, Jul 30, 2013 at 03:39:32PM +0100, Dominic Cleal wrote: > > Hmm why not. What are your proposals then? But I'd like to put tito > > machinery there. > > That's unacceptable, plugin authors may not be able to use tito. > > I'd suggest either keeping the spec file within the plugin repo itself, > or a repo like thirdparty within the 'theforeman' organisation that all > plugin maintainers have commit access to (though that's not perfect). > Builds could be triggered through Jenkins to avoid the need for tito, > koji etc for plugin maintainers?


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Actually, or not. I'm remembering now the conversation we had yesterday.

We could build the RPMs differently, similar to how thirdparty works in
that the plugin maintainer makes a release of the gem to rubygems.org,
and then we have a separate repo which contains all of the spec files
and .gem artifacts used for building the plugin RPMs.

I think though, I'd prefer a single release procedure rather than
separating the steps.

··· On 28/06/13 11:13, Dominic Cleal wrote: > On 28/06/13 11:06, Greg Sutcliffe wrote: >> Broadly, looks good. >> >> Packaging from a Debian standpoint isn't too hard (main issue will be >> correctly dropping new cache gems into vendor/cache, but this is my >> problem ;P), but I'm unsure where we should store the debian/ subdir >> which holds the packaging. You don't say where the Spec would live for >> the RPM either, so I'm no wiser on this. > > The spec would have to live in the plugin repo.


Dominic Cleal
Red Hat Engineering

> From: "Lukas Zapletal" <lzap@redhat.com>
> To: foreman-dev@googlegroups.com
> Sent: Tuesday, July 30, 2013 2:26:34 PM
> Subject: Re: [foreman-dev] Packaging of plugins
>
> > > Hmm why not. What are your proposals then? But I'd like to put tito
> > > machinery there.
> >
> > That's unacceptable, plugin authors may not be able to use tito.
> >
> > I'd suggest either keeping the spec file within the plugin repo itself,
> > or a repo like thirdparty within the 'theforeman' organisation that all
> > plugin maintainers have commit access to (though that's not perfect).
> > Builds could be triggered through Jenkins to avoid the need for tito,
> > koji etc for plugin maintainers?
>
> I am afraid plugin authors are not interested in packaging. These are
> two different things. We should keep in mind that packagers (that's
> mostly us) should be in our major focus. We should make this easy for
> packagers, not plugin authors.

I disagree with this approach. If we make it hard for them to get involved they will obviously never care. If the process is straightforward they will realize that and get involved.

>
> We must keep building in Koji. This is not about a gem or two, all these
> plugins will have dependencies and we need to package them, put them in
> repos and distribute them. We cannot make this simple process and avoid
> tito or koji at all.

We can absolutely avoid tito. I agree that we need to use Koji, though. That being said, plugin maintainers can still help maintain their own specs and build locally.

>
> Here are my preferences:
>
> A) Let plugin authors to do rubygems releasing, doing our packaging work
> using normal means - thirdparty repo, tito, koji, standard processes.

Thirdparty is not publicly available so that won't work. I don't understand why this can't be handled in an upstream git repo.

>
> B) Do the same, but creating new github repository, still using tito,
> koji, standard process. For each distribution type to create one
> subdirectory (debian, redhat) and making these two separate sandboxes.

I still don't like the fact that tito is involved, but I'd refer this option.

··· ----- Original Message ----- > On Tue, Jul 30, 2013 at 03:39:32PM +0100, Dominic Cleal wrote:


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sure, that's what I suggest - to use best tools to make packaging
easier. Putting a SPEC file in the repo with plugin does not make
anything easier at all, it's the other way around.

Where do you want to have SPEC file dependencies in the approach you and
Dom suggest?

LZ

··· On Tue, Jul 30, 2013 at 02:32:55PM -0400, Sam Kottler wrote: > I disagree with this approach. If we make it hard for them to get > involved they will obviously never care. If the process is > straightforward they will realize that and get involved.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Sounds reasonable. Tangentially: we probably should advertise Foreman's
anticipated release dates to make it easier for plugin authors/maintainers
to sync their schedules with ours.
-d

··· On Fri, Jun 28, 2013 at 11:18 AM, Dominic Cleal wrote:

On 28/06/13 11:13, Dominic Cleal wrote:

On 28/06/13 11:06, Greg Sutcliffe wrote:

Broadly, looks good.

Packaging from a Debian standpoint isn’t too hard (main issue will be
correctly dropping new cache gems into vendor/cache, but this is my
problem ;P), but I’m unsure where we should store the debian/ subdir
which holds the packaging. You don’t say where the Spec would live for
the RPM either, so I’m no wiser on this.

The spec would have to live in the plugin repo.

Actually, or not. I’m remembering now the conversation we had yesterday.

We could build the RPMs differently, similar to how thirdparty works in
that the plugin maintainer makes a release of the gem to rubygems.org,
and then we have a separate repo which contains all of the spec files
and .gem artifacts used for building the plugin RPMs.

I think though, I’d prefer a single release procedure rather than
separating the steps.


Dominic Cleal
Red Hat Engineering


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

> From: "Lukas Zapletal" <lzap@redhat.com>
> To: foreman-dev@googlegroups.com
> Sent: Tuesday, July 30, 2013 2:45:42 PM
> Subject: Re: [foreman-dev] Packaging of plugins
>
> > I disagree with this approach. If we make it hard for them to get
> > involved they will obviously never care. If the process is
> > straightforward they will realize that and get involved.
>
> Sure, that's what I suggest - to use best tools to make packaging
> easier. Putting a SPEC file in the repo with plugin does not make
> anything easier at all, it's the other way around.
>
> Where do you want to have SPEC file dependencies in the approach you and
> Dom suggest?

In the foreman-packaging repo.

··· ----- Original Message ----- > On Tue, Jul 30, 2013 at 02:32:55PM -0400, Sam Kottler wrote:

LZ


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

We've stated we want to release 3-4 times a year, and our RC phases are
usually 1-2 months long. That's enough time isn't it?

··· On 28 June 2013 13:10, Dmitri Dolguikh wrote:

Sounds reasonable. Tangentially: we probably should advertise Foreman’s
anticipated release dates to make it easier for plugin authors/maintainers
to sync their schedules with ours.
-d