Katello & Foreman Packaging

Development
1

I thought it'd be a good time to mention this again afer the thread
about hammer_cli_foreman_tasks, which was added to the Katello repos.

I think Katello people seem to prefer the way we're doing things, but
I'm not sure I understand. That's probably a product of having done far
more foreman packaging than for Katello, but…

hammer_cli_foreman_tasks has it's own rel-eng setup, and stores the
.spec file locally in its repo. Howiever, the comps are stored in the
Katello repo - which other than the fact Katello uses it, makes no sense
to me. A change, or adding a new package could require commits to
multiple repos.

Is there a reason we can't adopt foreman-packaging or our own
katello-packaging? Nobody is left wondering what to update, or where a
spec file is, and also - if a particular package update breaks Katello.

I think that last one is important.

If we were to centralize packaging all in one place for both foreman and
katello, we could ensure the Katello test suite is run whenever our
dependencies are updated, and we'd catch the problem of Foreman making
changes to packages that Katello uses much quicker.

I realize that's increasing the load on our already taxed Rackspace
budget, but the amount of time wasted when deps break Katello is really
far too much.

Anyway, just a thought…

··· -- Best Regards,

Stephen Benjamin
Red Hat Engineering

2

Now having done a few releases of a plugin myself through this workflow, I
tend to agree. The added bonus is that spec files are kept in sync and when
a change happens there is less mystery as to why a particular package
stopped building. For example, when the macro for plugins got moved to
foreman-plugin package all plugins in foreman-packaging got updated
properly. Further, this would provide less variance in how the specs are
defined and in the workflows that are used to release and build. I find
that this also tends to enforce thinking about actual releases - instead of
just releasing a new package, a gem is released properly to the ruby
community and then RPM release done based off of that.

The argument, as far as I can recall, against this has always been that you
can't search the repository for the tags directly and that you cannot
simply clone a repository and do a scratch builds or release. But I defer
to others who hold this opinion.

Eric

··· On Thu, Mar 5, 2015 at 5:30 AM, Stephen Benjamin wrote:

I thought it’d be a good time to mention this again afer the thread
about hammer_cli_foreman_tasks, which was added to the Katello repos.

I think Katello people seem to prefer the way we’re doing things, but
I’m not sure I understand. That’s probably a product of having done far
more foreman packaging than for Katello, but…

hammer_cli_foreman_tasks has it’s own rel-eng setup, and stores the
.spec file locally in its repo. Howiever, the comps are stored in the
Katello repo - which other than the fact Katello uses it, makes no sense
to me. A change, or adding a new package could require commits to
multiple repos.

Is there a reason we can’t adopt foreman-packaging or our own
katello-packaging? Nobody is left wondering what to update, or where a
spec file is, and also - if a particular package update breaks Katello.

I think that last one is important.

If we were to centralize packaging all in one place for both foreman and
katello, we could ensure the Katello test suite is run whenever our
dependencies are updated, and we’d catch the problem of Foreman making
changes to packages that Katello uses much quicker.

I realize that’s increasing the load on our already taxed Rackspace
budget, but the amount of time wasted when deps break Katello is really
far too much.

Anyway, just a thought…

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

> Now having done a few releases of a plugin myself through this workflow,
> I tend to agree. The added bonus is that spec files are kept in sync and
> when a change happens there is less mystery as to why a particular
> package stopped building. For example, when the macro for plugins got
> moved to foreman-plugin package all plugins in foreman-packaging got
> updated properly. Further, this would provide less variance in how the
> specs are defined and in the workflows that are used to release and
> build.

Thanks both for sharing your feedback. I think it's increasingly
important, as packagers tend to be different to developers - especially
when you consider Foreman is packaged for six+ OSes.

Having to maintain packaging inside disparate project repos makes it
more difficult to keep consistent as if I were to change something, I'd
have to interact with a dozen different plugin authors, many of whom
have no familiarity with packaging beyond the native format (gem).

I have no idea how I would orchestrate a mass change to something like
software collections if it wasn't centralised.

> I find that this also tends to enforce thinking about actual
> releases - instead of just releasing a new package, a gem is released
> properly to the ruby community and then RPM release done based off of that.

This is also the reason I've not dived into continuous/nightly builds of
plugins and have been watching for feedback. It gives us better parity
between packaged versions and gems available to developers working on
layered plugins who aren't using packages anyway.

I also want to provide a carrot to plugin developers to make regular,
stable updates to their plugins - releasing early, releasing often -
getting plugin updates to users sooner and not following Foreman's
quarterly cycle so tightly.

I think it's similar to an upstream and distro relationship, where we
try to ensure that our "distro" of >180 packages work well together.

> The argument, as far as I can recall, against this has always been that
> you can't search the repository for the tags directly and that you
> cannot simply clone a repository and do a scratch builds or release. But
> I defer to others who hold this opinion.

Yeah, it's harder to make a scratch build of a non-release. lzap wrote
an 'sbu' script (a simple wrapper around rpmbuild, mock etc) which can
be helpful, but there's always more we can do inside the packaging repo
to help with this with better READMEs through to custom tito builders.

··· On 05/03/15 15:42, Eric D Helms wrote:

–
Dominic Cleal
Red Hat Engineering

4

If you want to switch to a similar model as foreman-packaging, I think
I'd prefer to see a katello-packaging due to the large number of
packages you'd be adding and because they're separate repos.

··· On 05/03/15 10:30, Stephen Benjamin wrote: > I thought it'd be a good time to mention this again afer the thread > about hammer_cli_foreman_tasks, which was added to the Katello repos. > > I think Katello people seem to prefer the way we're doing things, but > I'm not sure I understand. That's probably a product of having done far > more foreman packaging than for Katello, but... > > hammer_cli_foreman_tasks has it's own rel-eng setup, and stores the > .spec file locally in its repo. Howiever, the comps are stored in the > Katello repo - which other than the fact Katello uses it, makes no sense > to me. A change, or adding a new package could require commits to > multiple repos. > > Is there a reason we can't adopt foreman-packaging or our own > katello-packaging?

–
Dominic Cleal
Red Hat Engineering

5

`

··· On 03/05/2015 12:29 PM, Dominic Cleal wrote: > On 05/03/15 15:42, Eric D Helms wrote: >> Now having done a few releases of a plugin myself through this workflow, >> I tend to agree. The added bonus is that spec files are kept in sync and >> when a change happens there is less mystery as to why a particular >> package stopped building. For example, when the macro for plugins got >> moved to foreman-plugin package all plugins in foreman-packaging got >> updated properly. Further, this would provide less variance in how the >> specs are defined and in the workflows that are used to release and >> build. > > Thanks both for sharing your feedback. I think it's increasingly > important, as packagers tend to be different to developers - especially > when you consider Foreman is packaged for six+ OSes. > > Having to maintain packaging inside disparate project repos makes it > more difficult to keep consistent as if I were to change something, I'd > have to interact with a dozen different plugin authors, many of whom > have no familiarity with packaging beyond the native format (gem). > > I have no idea how I would orchestrate a mass change to something like > software collections if it wasn't centralised. > >> I find that this also tends to enforce thinking about actual >> releases - instead of just releasing a new package, a gem is released >> properly to the ruby community and then RPM release done based off of that. > > This is also the reason I've not dived into continuous/nightly builds of > plugins and have been watching for feedback. It gives us better parity > between packaged versions and gems available to developers working on > layered plugins who aren't using packages anyway. > > I also want to provide a carrot to plugin developers to make regular, > stable updates to their plugins - releasing early, releasing often - > getting plugin updates to users sooner and not following Foreman's > quarterly cycle so tightly. > > I think it's similar to an upstream and distro relationship, where we > try to ensure that our "distro" of >180 packages work well together. > >> The argument, as far as I can recall, against this has always been that >> you can't search the repository for the tags directly and that you >> cannot simply clone a repository and do a scratch builds or release. But >> I defer to others who hold this opinion. > > Yeah, it's harder to make a scratch build of a non-release. lzap wrote > an 'sbu' script (a simple wrapper around rpmbuild, mock etc) which can > be helpful, but there's always more we can do inside the packaging repo > to help with this with better READMEs through to custom tito builders. >

I like stephens goal of getting more integrted builds up stream.
Stephen, what are the next steps to get there?

– bk

6

> `
>
> >>Now having done a few releases of a plugin myself through this workflow,
> >>I tend to agree. The added bonus is that spec files are kept in sync and
> >>when a change happens there is less mystery as to why a particular
> >>package stopped building. For example, when the macro for plugins got
> >>moved to foreman-plugin package all plugins in foreman-packaging got
> >>updated properly. Further, this would provide less variance in how the
> >>specs are defined and in the workflows that are used to release and
> >>build.
> >
> >Thanks both for sharing your feedback. I think it's increasingly
> >important, as packagers tend to be different to developers - especially
> >when you consider Foreman is packaged for six+ OSes.
> >
> >Having to maintain packaging inside disparate project repos makes it
> >more difficult to keep consistent as if I were to change something, I'd
> >have to interact with a dozen different plugin authors, many of whom
> >have no familiarity with packaging beyond the native format (gem).
> >
> >I have no idea how I would orchestrate a mass change to something like
> >software collections if it wasn't centralised.
> >
> >>I find that this also tends to enforce thinking about actual
> >>releases - instead of just releasing a new package, a gem is released
> >>properly to the ruby community and then RPM release done based off of that.
> >
> >This is also the reason I've not dived into continuous/nightly builds of
> >plugins and have been watching for feedback. It gives us better parity
> >between packaged versions and gems available to developers working on
> >layered plugins who aren't using packages anyway.
> >
> >I also want to provide a carrot to plugin developers to make regular,
> >stable updates to their plugins - releasing early, releasing often -
> >getting plugin updates to users sooner and not following Foreman's
> >quarterly cycle so tightly.
> >
> >I think it's similar to an upstream and distro relationship, where we
> >try to ensure that our "distro" of >180 packages work well together.
> >
> >>The argument, as far as I can recall, against this has always been that
> >>you can't search the repository for the tags directly and that you
> >>cannot simply clone a repository and do a scratch builds or release. But
> >>I defer to others who hold this opinion.
> >
> >Yeah, it's harder to make a scratch build of a non-release. lzap wrote
> >an 'sbu' script (a simple wrapper around rpmbuild, mock etc) which can
> >be helpful, but there's always more we can do inside the packaging repo
> >to help with this with better READMEs through to custom tito builders.
> >
>
> I like stephens goal of getting more integrted builds up stream. Stephen,
> what are the next steps to get there?

I think, to start, it's as simple as getting the comps setup for Katello
repos and start moving things over as we can.

I've opened an issue -
Feature #9660: As a developer, I want to manage katello packaging in the foreman-packaging repo - Packaging - Foreman

If we're all agreed on this, maybe we can start once 1.8/2.2 are out the
door and try to have it completed for 2.3.

··· On Thu, Mar 05, 2015 at 01:43:49PM -0500, Bryan Kearney wrote: > On 03/05/2015 12:29 PM, Dominic Cleal wrote: > >On 05/03/15 15:42, Eric D Helms wrote:

– bk

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

7

>
> > I thought it'd be a good time to mention this again afer the thread
> > about hammer_cli_foreman_tasks, which was added to the Katello repos.
> >
> > I think Katello people seem to prefer the way we're doing things, but
> > I'm not sure I understand. That's probably a product of having done far
> > more foreman packaging than for Katello, but…
> >
> > hammer_cli_foreman_tasks has it's own rel-eng setup, and stores the
> > .spec file locally in its repo. Howiever, the comps are stored in the
> > Katello repo - which other than the fact Katello uses it, makes no sense
> > to me. A change, or adding a new package could require commits to
> > multiple repos.
> >
> > Is there a reason we can't adopt foreman-packaging or our own
> > katello-packaging?
>
> If you want to switch to a similar model as foreman-packaging, I think
> I'd prefer to see a katello-packaging due to the large number of
> packages you'd be adding and because they're separate repos.
>

If we went our own repo route would we maintain the benefits of a
centralized packaging repository? Things like many eyes reviewing and
seeing changes (not just those of the katello team), PR testing, rel-eng
configurations, across the board spec updates such as when macros change.
If and when Katello becomes a properly installable plugin onto an existing
Foreman what happens ? Does it continue to live outside the plugins repo?

Eric

> –
> Dominic Cleal
> Red Hat Engineering
>
> –
> You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.

··· On Mar 6, 2015 3:10 AM, "Dominic Cleal" wrote: > On 05/03/15 10:30, Stephen Benjamin wrote: > For more options, visit https://groups.google.com/d/optout.
8

I would also want to understand what the policy is for a plugin to be
in foreman packaging vs not. If we all agree that katello should
follow the foreman standard policies, then lets know what they are for
all plugins and then move in that direction.

– bk

··· On 03/06/2015 07:13 AM, Eric D Helms wrote: > > On Mar 6, 2015 3:10 AM, "Dominic Cleal" > wrote: > > > > On 05/03/15 10:30, Stephen Benjamin wrote: > > > I thought it'd be a good time to mention this again afer the thread > > > about hammer_cli_foreman_tasks, which was added to the Katello repos. > > > > > > I think Katello people seem to prefer the way we're doing things, but > > > I'm not sure I understand. That's probably a product of having > done far > > > more foreman packaging than for Katello, but... > > > > > > hammer_cli_foreman_tasks has it's own rel-eng setup, and stores the > > > .spec file locally in its repo. Howiever, the comps are stored in the > > > Katello repo - which other than the fact Katello uses it, makes no > sense > > > to me. A change, or adding a new package could require commits to > > > multiple repos. > > > > > > Is there a reason we can't adopt foreman-packaging or our own > > > katello-packaging? > > > > If you want to switch to a similar model as foreman-packaging, I think > > I'd prefer to see a katello-packaging due to the large number of > > packages you'd be adding and because they're separate repos. > > > > If we went our own repo route would we maintain the benefits of a > centralized packaging repository? Things like many eyes reviewing and > seeing changes (not just those of the katello team), PR testing, rel-eng > configurations, across the board spec updates such as when macros > change. If and when Katello becomes a properly installable plugin onto > an existing Foreman what happens ? Does it continue to live outside the > plugins repo?
9

I thought before we decided one way or the other I would take the
rpm/develop branch of foreman-packaging and create the basic shell of what
a katello-packaging repository would look like. Thus, on my own fork of
foreman-packaging, I created a branch that removed all of the
foreman-specific packages and folder. Then I altered the comps and tito
configurations and opened a pull request to my branch to make the diff
easier to consume [1]. Two things I noticed that we'd have to resolve first:

  1. Create nonscl tags for each of the rhel 5/6/7 tags and adjust packages
    accordingly as well as the tag inheritance
  2. Store source tarballs somewhere for non-gem based packages (see example
    at [2])

Once those are resolved we could add packages to the repository 1 by 1
either by moving their building officially to that repository or by just
adding specs and configurations there till we have it entirely built out to
stand back and look at the result.

Eric

[1] https://github.com/ehelms/foreman-packaging/pull/1
[2]
https://github.com/theforeman/foreman-packaging/blob/rpm/develop/foreman/foreman.spec#L29

··· On Fri, Mar 6, 2015 at 8:02 AM, Bryan Kearney wrote:

On 03/06/2015 07:13 AM, Eric D Helms wrote:

On Mar 6, 2015 3:10 AM, “Dominic Cleal” <dcleal+g@redhat.com >> mailto:dcleal%2Bg@redhat.com> wrote:

On 05/03/15 10:30, Stephen Benjamin wrote:

I thought it’d be a good time to mention this again afer the thread
about hammer_cli_foreman_tasks, which was added to the Katello repos.

I think Katello people seem to prefer the way we’re doing things, but
I’m not sure I understand. That’s probably a product of having
done far

more foreman packaging than for Katello, but…

hammer_cli_foreman_tasks has it’s own rel-eng setup, and stores the
.spec file locally in its repo. Howiever, the comps are stored in
the

Katello repo - which other than the fact Katello uses it, makes no
sense

to me. A change, or adding a new package could require commits to
multiple repos.

Is there a reason we can’t adopt foreman-packaging or our own
katello-packaging?

If you want to switch to a similar model as foreman-packaging, I think
I’d prefer to see a katello-packaging due to the large number of
packages you’d be adding and because they’re separate repos.

If we went our own repo route would we maintain the benefits of a
centralized packaging repository? Things like many eyes reviewing and
seeing changes (not just those of the katello team), PR testing, rel-eng
configurations, across the board spec updates such as when macros
change. If and when Katello becomes a properly installable plugin onto
an existing Foreman what happens ? Does it continue to live outside the
plugins repo?

I would also want to understand what the policy is for a plugin to be in
foreman packaging vs not. If we all agree that katello should follow the
foreman standard policies, then lets know what they are for all plugins and
then move in that direction.

– bk

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

> > `
> >
> > >>Now having done a few releases of a plugin myself through this workflow,
> > >>I tend to agree. The added bonus is that spec files are kept in sync and
> > >>when a change happens there is less mystery as to why a particular
> > >>package stopped building. For example, when the macro for plugins got
> > >>moved to foreman-plugin package all plugins in foreman-packaging got
> > >>updated properly. Further, this would provide less variance in how the
> > >>specs are defined and in the workflows that are used to release and
> > >>build.
> > >
> > >Thanks both for sharing your feedback. I think it's increasingly
> > >important, as packagers tend to be different to developers - especially
> > >when you consider Foreman is packaged for six+ OSes.
> > >
> > >Having to maintain packaging inside disparate project repos makes it
> > >more difficult to keep consistent as if I were to change something, I'd
> > >have to interact with a dozen different plugin authors, many of whom
> > >have no familiarity with packaging beyond the native format (gem).
> > >
> > >I have no idea how I would orchestrate a mass change to something like
> > >software collections if it wasn't centralised.
> > >
> > >>I find that this also tends to enforce thinking about actual
> > >>releases - instead of just releasing a new package, a gem is released
> > >>properly to the ruby community and then RPM release done based off of
> > >>that.
> > >
> > >This is also the reason I've not dived into continuous/nightly builds of
> > >plugins and have been watching for feedback. It gives us better parity
> > >between packaged versions and gems available to developers working on
> > >layered plugins who aren't using packages anyway.
> > >
> > >I also want to provide a carrot to plugin developers to make regular,
> > >stable updates to their plugins - releasing early, releasing often -
> > >getting plugin updates to users sooner and not following Foreman's
> > >quarterly cycle so tightly.
> > >
> > >I think it's similar to an upstream and distro relationship, where we
> > >try to ensure that our "distro" of >180 packages work well together.
> > >
> > >>The argument, as far as I can recall, against this has always been that
> > >>you can't search the repository for the tags directly and that you
> > >>cannot simply clone a repository and do a scratch builds or release. But
> > >>I defer to others who hold this opinion.
> > >
> > >Yeah, it's harder to make a scratch build of a non-release. lzap wrote
> > >an 'sbu' script (a simple wrapper around rpmbuild, mock etc) which can
> > >be helpful, but there's always more we can do inside the packaging repo
> > >to help with this with better READMEs through to custom tito builders.
> > >
> >
> > I like stephens goal of getting more integrted builds up stream. Stephen,
> > what are the next steps to get there?
>
> I think, to start, it's as simple as getting the comps setup for Katello
> repos and start moving things over as we can.
>
> I've opened an issue -
> Feature #9660: As a developer, I want to manage katello packaging in the foreman-packaging repo - Packaging - Foreman
>
> If we're all agreed on this, maybe we can start once 1.8/2.2 are out the
> door and try to have it completed for 2.3.

i would prefer to learn more about how foreman-packaging works and of it intricacies, before i agree to anything.

··· ----- Original Message ----- > On Thu, Mar 05, 2015 at 01:43:49PM -0500, Bryan Kearney wrote: > > On 03/05/2015 12:29 PM, Dominic Cleal wrote: > > >On 05/03/15 15:42, Eric D Helms wrote:

– bk

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–

  • adam price
11

>
>
> > > `
> > >
> > > >>Now having done a few releases of a plugin myself through this workflow,
> > > >>I tend to agree. The added bonus is that spec files are kept in sync and
> > > >>when a change happens there is less mystery as to why a particular
> > > >>package stopped building. For example, when the macro for plugins got
> > > >>moved to foreman-plugin package all plugins in foreman-packaging got
> > > >>updated properly. Further, this would provide less variance in how the
> > > >>specs are defined and in the workflows that are used to release and
> > > >>build.
> > > >
> > > >Thanks both for sharing your feedback. I think it's increasingly
> > > >important, as packagers tend to be different to developers - especially
> > > >when you consider Foreman is packaged for six+ OSes.
> > > >
> > > >Having to maintain packaging inside disparate project repos makes it
> > > >more difficult to keep consistent as if I were to change something, I'd
> > > >have to interact with a dozen different plugin authors, many of whom
> > > >have no familiarity with packaging beyond the native format (gem).
> > > >
> > > >I have no idea how I would orchestrate a mass change to something like
> > > >software collections if it wasn't centralised.
> > > >
> > > >>I find that this also tends to enforce thinking about actual
> > > >>releases - instead of just releasing a new package, a gem is released
> > > >>properly to the ruby community and then RPM release done based off of
> > > >>that.
> > > >
> > > >This is also the reason I've not dived into continuous/nightly builds of
> > > >plugins and have been watching for feedback. It gives us better parity
> > > >between packaged versions and gems available to developers working on
> > > >layered plugins who aren't using packages anyway.
> > > >
> > > >I also want to provide a carrot to plugin developers to make regular,
> > > >stable updates to their plugins - releasing early, releasing often -
> > > >getting plugin updates to users sooner and not following Foreman's
> > > >quarterly cycle so tightly.
> > > >
> > > >I think it's similar to an upstream and distro relationship, where we
> > > >try to ensure that our "distro" of >180 packages work well together.
> > > >
> > > >>The argument, as far as I can recall, against this has always been that
> > > >>you can't search the repository for the tags directly and that you
> > > >>cannot simply clone a repository and do a scratch builds or release. But
> > > >>I defer to others who hold this opinion.
> > > >
> > > >Yeah, it's harder to make a scratch build of a non-release. lzap wrote
> > > >an 'sbu' script (a simple wrapper around rpmbuild, mock etc) which can
> > > >be helpful, but there's always more we can do inside the packaging repo
> > > >to help with this with better READMEs through to custom tito builders.
> > > >
> > >
> > > I like stephens goal of getting more integrted builds up stream. Stephen,
> > > what are the next steps to get there?
> >
> > I think, to start, it's as simple as getting the comps setup for Katello
> > repos and start moving things over as we can.
> >
> > I've opened an issue -
> > Feature #9660: As a developer, I want to manage katello packaging in the foreman-packaging repo - Packaging - Foreman
> >
> > If we're all agreed on this, maybe we can start once 1.8/2.2 are out the
> > door and try to have it completed for 2.3.
>
> i would prefer to learn more about how foreman-packaging works and of it intricacies, before i agree to anything.

That's fair.

The readme has some info:
https://github.com/theforeman/foreman-packaging/tree/rpm/develop

And you can take a look at the PR's to get an idea of what it looks
like. Here's an example of adding a new package, hammer_cli_foreman_salt:
https://github.com/theforeman/foreman-packaging/pull/495

··· On Thu, Mar 05, 2015 at 03:30:45PM -0500, Adam Price wrote: > ----- Original Message ----- > > On Thu, Mar 05, 2015 at 01:43:49PM -0500, Bryan Kearney wrote: > > > On 03/05/2015 12:29 PM, Dominic Cleal wrote: > > > >On 05/03/15 15:42, Eric D Helms wrote:

– bk

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–

  • adam price

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

12

I have been diving into this a bit over the past week when I have had time
in order to get used to trying to convert and identify any pain points. I
switched gears and created [1] as a test ground and opened a couple of PRs
to add initial basic configuration and 1-2 package conversions. There are 2
areas of concern that give me pause:

  1. Duplication

For parity, we will need to create at least two Jenkins jobs to mirror
'packaging_test_pull_request_rpm' and 'packaging_build_rpm' and update
foreman-infra to include our new repository plus hook it up to the
prprocessor. This could potentially be minimized somewhat by converting
those jobs to JJB first and re-using scripts.

  1. Linus' Law

I worry that by having two separate repositories we are asking those of us
that know packaging to monitor two separate repositories to give input and
review. Or, that we will naturally focus on one or the other and the
packaging knowledge we do have won't end up shared across the space and
quality will suffer. One option for this could be to start with Katello
branches on foreman-packaging that way we are at least a centralized
repository without imposing all of the Katello dependencies (it is my hope
that these dwindle over time with some of the upcoming features) onto the
stable Foreman packaging branches.

Eric

[1] https://github.com/Katello/katello-packaging

··· On Wed, Apr 15, 2015 at 9:18 PM, Eric D Helms wrote:

I thought before we decided one way or the other I would take the
rpm/develop branch of foreman-packaging and create the basic shell of what
a katello-packaging repository would look like. Thus, on my own fork of
foreman-packaging, I created a branch that removed all of the
foreman-specific packages and folder. Then I altered the comps and tito
configurations and opened a pull request to my branch to make the diff
easier to consume [1]. Two things I noticed that we’d have to resolve first:

  1. Create nonscl tags for each of the rhel 5/6/7 tags and adjust packages
    accordingly as well as the tag inheritance
  2. Store source tarballs somewhere for non-gem based packages (see example
    at [2])

Once those are resolved we could add packages to the repository 1 by 1
either by moving their building officially to that repository or by just
adding specs and configurations there till we have it entirely built out to
stand back and look at the result.

Eric

[1] https://github.com/ehelms/foreman-packaging/pull/1
[2]
https://github.com/theforeman/foreman-packaging/blob/rpm/develop/foreman/foreman.spec#L29

On Fri, Mar 6, 2015 at 8:02 AM, Bryan Kearney bryan.kearney@gmail.com > wrote:

On 03/06/2015 07:13 AM, Eric D Helms wrote:

On Mar 6, 2015 3:10 AM, “Dominic Cleal” <dcleal+g@redhat.com >>> mailto:dcleal%2Bg@redhat.com> wrote:

On 05/03/15 10:30, Stephen Benjamin wrote:

I thought it’d be a good time to mention this again afer the thread
about hammer_cli_foreman_tasks, which was added to the Katello
repos.

I think Katello people seem to prefer the way we’re doing things,
but

I’m not sure I understand. That’s probably a product of having
done far

more foreman packaging than for Katello, but…

hammer_cli_foreman_tasks has it’s own rel-eng setup, and stores the
.spec file locally in its repo. Howiever, the comps are stored in
the

Katello repo - which other than the fact Katello uses it, makes no
sense

to me. A change, or adding a new package could require commits to
multiple repos.

Is there a reason we can’t adopt foreman-packaging or our own
katello-packaging?

If you want to switch to a similar model as foreman-packaging, I think
I’d prefer to see a katello-packaging due to the large number of
packages you’d be adding and because they’re separate repos.

If we went our own repo route would we maintain the benefits of a
centralized packaging repository? Things like many eyes reviewing and
seeing changes (not just those of the katello team), PR testing, rel-eng
configurations, across the board spec updates such as when macros
change. If and when Katello becomes a properly installable plugin onto
an existing Foreman what happens ? Does it continue to live outside the
plugins repo?

I would also want to understand what the policy is for a plugin to be
in foreman packaging vs not. If we all agree that katello should follow
the foreman standard policies, then lets know what they are for all plugins
and then move in that direction.

– bk

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

13

Prompted by GitLab's announcement of support for git annex I started
looking at using foreman-packaging to build dependencies downstream. Build
dependencies directly from foreman-packaging is a drastic improvement over
our previous setup. However, once moving beyond Foreman dependencies things
get really murky as to where source and rpm specs are stored. It would be
great to have a very similar setup for katello dependencies - whether that
is foreman-packaging or katello-packaging wouldn't matter much to me, but
having just one or two places to go to builds some high percentage of
dependencies would be great.

··· On Thursday, March 5, 2015 at 4:21:52 PM UTC-5, Stephen Benjamin wrote: > > On Thu, Mar 05, 2015 at 03:30:45PM -0500, Adam Price wrote: > > > > > > ----- Original Message ----- > > > On Thu, Mar 05, 2015 at 01:43:49PM -0500, Bryan Kearney wrote: > > > > ` > > > > > > > > On 03/05/2015 12:29 PM, Dominic Cleal wrote: > > > > >On 05/03/15 15:42, Eric D Helms wrote: > > > > >>Now having done a few releases of a plugin myself through this > workflow, > > > > >>I tend to agree. The added bonus is that spec files are kept in > sync and > > > > >>when a change happens there is less mystery as to why a particular > > > > >>package stopped building. For example, when the macro for plugins > got > > > > >>moved to foreman-plugin package all plugins in foreman-packaging > got > > > > >>updated properly. Further, this would provide less variance in how > the > > > > >>specs are defined and in the workflows that are used to release > and > > > > >>build. > > > > > > > > > >Thanks both for sharing your feedback. I think it's increasingly > > > > >important, as packagers tend to be different to developers - > especially > > > > >when you consider Foreman is packaged for six+ OSes. > > > > > > > > > >Having to maintain packaging inside disparate project repos makes > it > > > > >more difficult to keep consistent as if I were to change something, > I'd > > > > >have to interact with a dozen different plugin authors, many of > whom > > > > >have no familiarity with packaging beyond the native format (gem). > > > > > > > > > >I have no idea how I would orchestrate a mass change to something > like > > > > >software collections if it wasn't centralised. > > > > > > > > > >>I find that this also tends to enforce thinking about actual > > > > >>releases - instead of just releasing a new package, a gem is > released > > > > >>properly to the ruby community and then RPM release done based off > of > > > > >>that. > > > > > > > > > >This is also the reason I've not dived into continuous/nightly > builds of > > > > >plugins and have been watching for feedback. It gives us better > parity > > > > >between packaged versions and gems available to developers working > on > > > > >layered plugins who aren't using packages anyway. > > > > > > > > > >I also want to provide a carrot to plugin developers to make > regular, > > > > >stable updates to their plugins - releasing early, releasing often > - > > > > >getting plugin updates to users sooner and not following Foreman's > > > > >quarterly cycle so tightly. > > > > > > > > > >I think it's similar to an upstream and distro relationship, where > we > > > > >try to ensure that our "distro" of >180 packages work well > together. > > > > > > > > > >>The argument, as far as I can recall, against this has always been > that > > > > >>you can't search the repository for the tags directly and that you > > > > >>cannot simply clone a repository and do a scratch builds or > release. But > > > > >>I defer to others who hold this opinion. > > > > > > > > > >Yeah, it's harder to make a scratch build of a non-release. lzap > wrote > > > > >an 'sbu' script (a simple wrapper around rpmbuild, mock etc) which > can > > > > >be helpful, but there's always more we can do inside the packaging > repo > > > > >to help with this with better READMEs through to custom tito > builders. > > > > > > > > > > > > > I like stephens goal of getting more integrted builds up stream. > Stephen, > > > > what are the next steps to get there? > > > > > > I think, to start, it's as simple as getting the comps setup for > Katello > > > repos and start moving things over as we can. > > > > > > I've opened an issue - > > > http://projects.theforeman.org/issues/9660 > > > > > > If we're all agreed on this, maybe we can start once 1.8/2.2 are out > the > > > door and try to have it completed for 2.3. > > > > i would prefer to learn more about how foreman-packaging works and of it > intricacies, before i agree to anything. > > That's fair. > > The readme has some info: > https://github.com/theforeman/foreman-packaging/tree/rpm/develop > > And you can take a look at the PR's to get an idea of what it looks > like. Here's an example of adding a new package, hammer_cli_foreman_salt: > https://github.com/theforeman/foreman-packaging/pull/495 > > > > > > > > > > > > > > > -- bk > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups > > > > "foreman-dev" group. > > > > To unsubscribe from this group and stop receiving emails from it, > send an > > > > email to foreman-dev...@googlegroups.com . > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > > Best Regards, > > > > > > Stephen Benjamin > > > Red Hat Engineering > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "foreman-dev" group. > > > To unsubscribe from this group and stop receiving emails from it, send > an > > > email to foreman-dev...@googlegroups.com . > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > -- > > - adam price > > > > -- > > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-dev...@googlegroups.com . > > For more options, visit https://groups.google.com/d/optout. > >
14

IMHO… the closer to get to one model, the better.

– bk

··· On 03/05/2015 04:59 PM, Jason Montleon wrote: > Prompted by GitLab's announcement of support for git annex I started > looking at using foreman-packaging to build dependencies downstream. > Build dependencies directly from foreman-packaging is a drastic > improvement over our previous setup. However, once moving beyond Foreman > dependencies things get really murky as to where source and rpm specs > are stored. It would be great to have a very similar setup for katello > dependencies - whether that is foreman-packaging or katello-packaging > wouldn't matter much to me, but having just one or two places to go to > builds some high percentage of dependencies would be great. > > On Thursday, March 5, 2015 at 4:21:52 PM UTC-5, Stephen Benjamin wrote: > > On Thu, Mar 05, 2015 at 03:30:45PM -0500, Adam Price wrote: > > > > > > ----- Original Message ----- > > > On Thu, Mar 05, 2015 at 01:43:49PM -0500, Bryan Kearney wrote: > > > > ` > > > > > > > > On 03/05/2015 12:29 PM, Dominic Cleal wrote: > > > > >On 05/03/15 15:42, Eric D Helms wrote: > > > > >>Now having done a few releases of a plugin myself through > this workflow, > > > > >>I tend to agree. The added bonus is that spec files are > kept in sync and > > > > >>when a change happens there is less mystery as to why a > particular > > > > >>package stopped building. For example, when the macro for > plugins got > > > > >>moved to foreman-plugin package all plugins in > foreman-packaging got > > > > >>updated properly. Further, this would provide less variance > in how the > > > > >>specs are defined and in the workflows that are used to > release and > > > > >>build. > > > > > > > > > >Thanks both for sharing your feedback. I think it's > increasingly > > > > >important, as packagers tend to be different to developers - > especially > > > > >when you consider Foreman is packaged for six+ OSes. > > > > > > > > > >Having to maintain packaging inside disparate project repos > makes it > > > > >more difficult to keep consistent as if I were to change > something, I'd > > > > >have to interact with a dozen different plugin authors, many > of whom > > > > >have no familiarity with packaging beyond the native format > (gem). > > > > > > > > > >I have no idea how I would orchestrate a mass change to > something like > > > > >software collections if it wasn't centralised. > > > > > > > > > >>I find that this also tends to enforce thinking about actual > > > > >>releases - instead of just releasing a new package, a gem > is released > > > > >>properly to the ruby community and then RPM release done > based off of > > > > >>that. > > > > > > > > > >This is also the reason I've not dived into > continuous/nightly builds of > > > > >plugins and have been watching for feedback. It gives us > better parity > > > > >between packaged versions and gems available to developers > working on > > > > >layered plugins who aren't using packages anyway. > > > > > > > > > >I also want to provide a carrot to plugin developers to make > regular, > > > > >stable updates to their plugins - releasing early, releasing > often - > > > > >getting plugin updates to users sooner and not following > Foreman's > > > > >quarterly cycle so tightly. > > > > > > > > > >I think it's similar to an upstream and distro relationship, > where we > > > > >try to ensure that our "distro" of >180 packages work well > together. > > > > > > > > > >>The argument, as far as I can recall, against this has > always been that > > > > >>you can't search the repository for the tags directly and > that you > > > > >>cannot simply clone a repository and do a scratch builds or > release. But > > > > >>I defer to others who hold this opinion. > > > > > > > > > >Yeah, it's harder to make a scratch build of a non-release. > lzap wrote > > > > >an 'sbu' script (a simple wrapper around rpmbuild, mock etc) > which can > > > > >be helpful, but there's always more we can do inside the > packaging repo > > > > >to help with this with better READMEs through to custom tito > builders. > > > > > > > > > > > > > I like stephens goal of getting more integrted builds up > stream. Stephen, > > > > what are the next steps to get there? > > > > > > I think, to start, it's as simple as getting the comps setup > for Katello > > > repos and start moving things over as we can. > > > > > > I've opened an issue - > > > http://projects.theforeman.org/issues/9660 > > > > > > > If we're all agreed on this, maybe we can start once 1.8/2.2 > are out the > > > door and try to have it completed for 2.3. > > > > i would prefer to learn more about how foreman-packaging works > and of it intricacies, before i agree to anything. > > That's fair. > > The readme has some info: > https://github.com/theforeman/foreman-packaging/tree/rpm/develop > > > And you can take a look at the PR's to get an idea of what it looks > like. Here's an example of adding a new package, > hammer_cli_foreman_salt: > https://github.com/theforeman/foreman-packaging/pull/495 > > > > > > > > > > > > > > > > -- bk > > > > > > > > -- > > > > You received this message because you are subscribed to the > Google Groups > > > > "foreman-dev" group. > > > > To unsubscribe from this group and stop receiving emails from > it, send an > > > > email to foreman-dev...@googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout > . > > > > > > -- > > > Best Regards, > > > > > > Stephen Benjamin > > > Red Hat Engineering > > > > > > -- > > > You received this message because you are subscribed to the > Google Groups > > > "foreman-dev" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an > > > email to foreman-dev...@googlegroups.com. > > > For more options, visit https://groups.google.com/d/optout > . > > > > > > > -- > > - adam price > > > > -- > > You received this message because you are subscribed to the > Google Groups "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to foreman-dev...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout > . > > -- > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-dev+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout.
15

If we were to switch to foreman-packaging, I think we could do so in
gradual way that could simultaneously result in cleaning up our
repositories of unused cruft. Before we did such a thing, I would
personally like to see:

  1. A migration strategy
  2. Acknowledgement and agreement from those that have historically and
    currently do packaging on the Katello side as well as that handle Foreman
    packaging, this includes but is not limited to: myself, Justin, Mike, Adam
    and Dominic
  3. A deep dive of foreman-packaging for those unfamiliar with the layout
    and process so that everyone understands the process by which things would
    occur moving forward.

I am more than willing to lend effort toward the end goal. For some
background and information, I'll try to give a lay of the land of
repositories and packages that we maintain currently as this may help to
understand the current scope and provide a starting point for discerning
the best strategy to move to foreman-packaging in a way that makes sense.
Please fill in blanks where I am missinganything.

Repositories

The repositories we maintain and publish with any release can be seen by
looking at the comps that we keep at [1]. The repositories and brief
explanations of each are:

  • Katello Server RHEL6/7
    • The Katello server packages, installer packages and thirdparty gems
      and libraries that Katello requires but are not available in either base
      OS, epel, SCL or Foreman repositories.
  • Katello Client RHEL 5/6/7
    • The Katello client packages for using katello-agent.
  • Katello Pulp 6/7
    • A generated repository of a specific Pulp version that is being used
      currently or with a particular version of Katello. This is so that we can
      control when Pulp is updated for Katello due to our tight integration. Note
      that this also contains qpid packages that are not specifically required by
      Pulp, but used by Katello/Pulp/Candlepin.
  • Katello Candlepin 6/7
    • A generated repository of a specific Candlepin version that is being
      used currently or with a particular version of Katello. This is so that we
      can control when Candlepin is updated for Katello due to our tight
      integration.

For Candlepin and Pulp, those teams control building and maintaining the
RPMs which are built into our Koji. The repositories we maintain are built
by tagging those builds for our purposes. That is to say, we do not
maintain nor handle packaging Pulp and Candlepin.

Packages

There are various packages that the Katello team maintains and versions
from release to release that are currently stored within their respective
git repositories. They are:

  • The Katello, katello-devel, capsule and SAM installer packages.
  • CLI module for Katello
  • OpenSSL wrapper for generating various certificate types as well as
    RPMs that wrap and deploy them. Heavily used by the puppet-certs module for
    all cert generation. Undergoes very minor changes these days.
  • Generates a single RPM users can install to configure the yum
    repositories for Katello and components.
  • This provides katello-disconnected functionality for syncing
    repositories and carrying them across an air gap to a Katello server.
  • CLI module for foreman_gutterball
  • CLI module for importing data from a Spacewalk/Satellite
  • All the selinux rules needed for Katello
  • Foreman and Katello plugin that turns a Foreman/katello into a SAM
  • A set of API bindings in Ruby for Pulp.

Thirdparty Packages

As well as our own projects and packages, there are some third party
libraries we currently carry in our repositories based on our nightly
release. There is a possibility we have missed some of these being moved
into a Foreman repository and no longer needing to be tagged in our nightly
or that we no longer need to provide.

EL6/7

  • elasticsearch
  • lucene
  • snappy-java
  • sigar
  • pulp-katello
  • rubygem-algebrick
  • rubygem-anemone
  • rubygem-ansi
  • rubygem-angular-rails-templates
  • rubygem-chunky-png
  • rubygem-daemons
  • rubygem-deface
  • rubygem-dynflow
  • rubygem-foreman-tasks
  • rubygem-fssm
  • rubygem-haml
  • rubygem-haml-rails
  • rubygem-hashr
  • rubygem-hooks
  • rubygem-hpricot
  • rubygem-i18n_data
  • rubygem-justified
  • rubygem-little-plugger
  • rubygem-qpid_messaging
  • rubygem-rake-compiler
  • rubygem-robotex
  • rubygem-sprockets
  • rubygem-sprockets-rails
  • rubygem-strong_parameters
  • rubygem-tire

EL6 Only

  • ruby-shadow
  • ruby-wrapper
  • rubygem-addressable
  • rubygem-Ascii85
  • rubygem-POpen4
  • rubygem-bundler
  • rubygem-bundler_ext
  • rubygem-closure-compiler
  • rubygem-color
  • rubygem-commonks
  • rubygem-crack
  • rubygem-delayed_job
  • rubygem-eventmachine
  • rubygem-fastercsv
  • rubygem-flexmock
  • rubygem-foreigner
  • rubygem-foreman_api
  • rubygem-hoe
  • rubygem-i18n_data
  • rubygem-logging
  • rubygem-oauth
  • rubygem-open4
  • rubygem-nokogiri
  • rubygem-pg
  • rubygem-rack-mount
  • rubygem-rails_warden
  • rubygem-rcov
  • rubygem-reign
  • rubygem-rest-client
  • rubygem-rspec-rails
  • rubygem-ruby-progressbar
  • rubygem-ruby
  • rubygem-rubyforge
  • rubygem-simplecov
  • rubygem-syntax
  • rubygem-text-format
  • rubygem-thin
  • rubygem-transaction-simple
  • rubygem-ttfunk
  • rubygem-webmock
  • rubygem-warden
  • rubygem-kafo
  • sigar
  • sigar-java

[1] https://github.com/Katello/katello/tree/master/rel-eng/comps

Eric

16

> I worry that by having two separate repositories we are asking those of
> us that know packaging to monitor two separate repositories to give
> input and review.

This works best for me, as I only want to manage and see Foreman things.

> One option for this could be to start
> with Katello branches on foreman-packaging that way we are at least a
> centralized repository without imposing all of the Katello dependencies
> (it is my hope that these dwindle over time with some of the upcoming
> features) onto the stable Foreman packaging branches.

Worst of both worlds: it increases the problems when working on multiple
things in a single repo, increases the noise for those of us already
working in this repo, and for little or no benefit. Please keep it
under the Katello project.

··· On 26/05/15 20:15, Eric D Helms wrote:

–
Dominic Cleal
Red Hat Engineering

17

What I would like to see is the shared code around the releasers and helper scripts, so
that thy work consistently, no matter if I build foreman or katello packages, for
example things like how to build scratches (that was just added to foreman-packaging)

– Ivan

··· ----- Original Message ----- > On 26/05/15 20:15, Eric D Helms wrote: > > I worry that by having two separate repositories we are asking those of > > us that know packaging to monitor two separate repositories to give > > input and review. > > This works best for me, as I only want to manage and see Foreman things. > > > One option for this could be to start > > with Katello branches on foreman-packaging that way we are at least a > > centralized repository without imposing all of the Katello dependencies > > (it is my hope that these dwindle over time with some of the upcoming > > features) onto the stable Foreman packaging branches. > > Worst of both worlds: it increases the problems when working on multiple > things in a single repo, increases the noise for those of us already > working in this repo, and for little or no benefit. Please keep it > under the Katello project. > > -- > Dominic Cleal > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >