Vendorizing or Building RPMs

Development
1

Note: This is specific to RPMs because as far as I know the Debian process
is different and uses gems directly. Please correct me and contribute any
information with respect to Debian that I miss. I believe the Nodejs part
of the email does apply to Debian.

This proposal is in two parts: gems and node packages. The general issue
applies to both, the project has a need to be able to move and update
dependencies more quickly given the speed at which platforms like Rails,
React and Webpack update.

Node/NPM Packages

Today, we package nearly all nodejs packages that are required to either
build the UI or runtime UI dependencies into packages. At current count,
there are 78 nodejs packages listed in Koji as RPMs. Over the past few
months, the move to React and updating UI within Foreman core and branching
into plugins has led to multiple, and at times, compounding packaging and
thus nightly breakages. One of those breakages lasted over a week with
multiple developers spending many hours tracking down the issue. The speed
at which these changes are made makes keeping up successful nightly builds
difficult. The git repository gets well ahead of the packaging which leads
to compounding issues. This has also been the cause of stable version
release delays.

Proposal:
Deprecate nodejs packages in favor of foreman-assets or a new RPM
foreman-node-modules that contains a source tarball of node_modules/
packaged into a simple RPM that is used as a build dependency. This tarball
would be regenerated, and the package bumped, as dependency updates are
needed.

Ruby Gems

Today we package all required rubygems as RPMs and utilize a thirdparty
Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack
(sclo-ror42). The team that manages the RHSCLs has already released Ruby
2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no
plans to release any further SCLs for the Rails stack. This means, in
addition to our own dependencies, we need to build and manage the Rails
stack for the versions we want. This adds more packaging burden on the RPM
side. GIven this, we have a few options:

  1. Build and manage our own SCLs for every version of Rails from here on
    out
  2. Vendorize Rails and thirdparty gems into a one or more base RPMs
    (e.g. an RPM for rails stack and one for thirdparty dependencies)

Option 2, to vendorize, is a deviation from our prior practices in the area
of production deployment. Thus, I am reaching out to the community to get
feedback. One interesting consideration for vendorizing is when containers
are considered and having the ability to build them using 'bundle install'
versus using RPM based installation.

In theory vendorizing allows the community to move faster, keep up to date
with dependencies more easily and reduces the burden on RPM packaging to
keep nightly builds flowing. However, this does stray from the standard RPM
based installation benefits typically associated to it.

Please consider carefully, and weigh in even if it is simply to +1 or -1 a
given idea. Deployment is a large part of our project and the more voices
weighing in the better.

Thanks,
Eriic

2

Hi,

> Node/NPM Packages
>
> […]
>
> Proposal:
> Deprecate nodejs packages in favor of foreman-assets or a new RPM
> foreman-node-modules that contains a source tarball of node_modules/
> packaged into a simple RPM that is used as a build dependency. This tarball
> would be regenerated, and the package bumped, as dependency updates are
> needed.

I do see a problem to apply this to Debian. There are "native
extensions" (or whatever is the correct term in the Node world), used
i.e. by node-sass that differ by architecture and Node API versionn.

Gor Debian we're building on various architectures and also it's not
100% sure the same Node version is used (esp. it's often a different
version than the one coming from EPEL).

However, I don't see a problem with just leaving the Debian build
process as (which is just a npm install at package build time).

> Today we package all required rubygems as RPMs and utilize a thirdparty
> Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack
> (sclo-ror42). The team that manages the RHSCLs has already released Ruby
> 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no
> plans to release any further SCLs for the Rails stack. This means, in
> addition to our own dependencies, we need to build and manage the Rails
> stack for the versions we want. This adds more packaging burden on the RPM
> side. GIven this, we have a few options:
>
> 1) Build and manage our own SCLs for every version of Rails from here on
> out
> 2) Vendorize Rails and thirdparty gems into a one or more base RPMs
> (e.g. an RPM for rails stack and one for thirdparty dependencies)

I don't really have stakes in RPMs, but I'd try to stick with option 1
and I even expect that there's some demand for Rails SCLs at least in
the RHEL/CentOS world by other projects, too.

Regards

··· On Sat, Oct 14, 2017 at 02:24:58PM -0400, Eric D Helms wrote: -- Michael Moll
3

> Note: This is specific to RPMs because as far as I know the Debian
> process is different and uses gems directly. Please correct me and
> contribute any information with respect to Debian that I miss. I
> believe the Nodejs part of the email does apply to Debian.

As Michael says, I think we can mostly leave Debs out, at least for
now.

> Node/NPM Packages … Proposal:
> Deprecate nodejs packages in favor of foreman-assets or a new RPM
> foreman-node-modules that contains a source tarball of node_modules/
> packaged into a simple RPM that is used as a build dependency. This
> tarball would be regenerated, and the package bumped, as dependency
> updates are needed.

If we go for option 2 below, then vendoring an npm install at build
time (as Michael said, we do this for Debian) feels like a better
option.

If we go for building our own SCL, then +1 for this.

> Ruby Gems
>
> Option 2, to vendorize, is a deviation from our prior practices in
> the area of production deployment. Thus, I am reaching out to the
> community to get feedback. One interesting consideration for
> vendorizing is when containers are considered and having the ability
> to build them using 'bundle install' versus using RPM based
> installation.

Vendoring hasn't (to my knowledge) caused many issues for Debian users
(Michael?), and having consistent build processes makes it easier for
anyone to support users on different OSs.

That said, I've not really been involved with the RPMs, so I'm unsure
if this causes a bigger headache for Yum users than Apt users. I'm also
unsure of the work required to create an SCL, but if it's non-trivial
then I'd be looking to CentOS to collaborate on a Rails SCL for
everyone to use - for just ourselves, then vendoring seems easier.

Greg

··· On Sat, 2017-10-14 at 14:24 -0400, Eric D Helms wrote:
4

I'm ok with the vendorizing approach, as long as it makes the release
process more
effective than it is right now.

Could you expand more on how the core dependencies, plugins and
plugin-dependencies work?
Perhaps this is something we could align with the debian packaging as well to
address some of the issues that we are facing.

– Ivan

··· On Sat, Oct 14, 2017 at 8:24 PM, Eric D Helms wrote: > Note: This is specific to RPMs because as far as I know the Debian process > is different and uses gems directly. Please correct me and contribute any > information with respect to Debian that I miss. I believe the Nodejs part of > the email does apply to Debian. > > This proposal is in two parts: gems and node packages. The general issue > applies to both, the project has a need to be able to move and update > dependencies more quickly given the speed at which platforms like Rails, > React and Webpack update. > > > Node/NPM Packages > > Today, we package nearly all nodejs packages that are required to either > build the UI or runtime UI dependencies into packages. At current count, > there are 78 nodejs packages listed in Koji as RPMs. Over the past few > months, the move to React and updating UI within Foreman core and branching > into plugins has led to multiple, and at times, compounding packaging and > thus nightly breakages. One of those breakages lasted over a week with > multiple developers spending many hours tracking down the issue. The speed > at which these changes are made makes keeping up successful nightly builds > difficult. The git repository gets well ahead of the packaging which leads > to compounding issues. This has also been the cause of stable version > release delays. > > Proposal: > Deprecate nodejs packages in favor of foreman-assets or a new RPM > foreman-node-modules that contains a source tarball of node_modules/ > packaged into a simple RPM that is used as a build dependency. This tarball > would be regenerated, and the package bumped, as dependency updates are > needed. > > > Ruby Gems > > Today we package all required rubygems as RPMs and utilize a thirdparty > Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack > (sclo-ror42). The team that manages the RHSCLs has already released Ruby 2.3 > and RUby 2.4 SCLs and will continue to do so. However, there are no plans to > release any further SCLs for the Rails stack. This means, in addition to our > own dependencies, we need to build and manage the Rails stack for the > versions we want. This adds more packaging burden on the RPM side. GIven > this, we have a few options: > > 1) Build and manage our own SCLs for every version of Rails from here on > out > 2) Vendorize Rails and thirdparty gems into a one or more base RPMs (e.g. > an RPM for rails stack and one for thirdparty dependencies) > > Option 2, to vendorize, is a deviation from our prior practices in the area > of production deployment. Thus, I am reaching out to the community to get > feedback. One interesting consideration for vendorizing is when containers > are considered and having the ability to build them using 'bundle install' > versus using RPM based installation. > > In theory vendorizing allows the community to move faster, keep up to date > with dependencies more easily and reduces the burden on RPM packaging to > keep nightly builds flowing. However, this does stray from the standard RPM > based installation benefits typically associated to it. > > > Please consider carefully, and weigh in even if it is simply to +1 or -1 a > given idea. Deployment is a large part of our project and the more voices > weighing in the better. > > Thanks, > Eriic > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.
5

>Note: This is specific to RPMs because as far as I know the Debian process
>is different and uses gems directly. Please correct me and contribute any
>information with respect to Debian that I miss. I believe the Nodejs part
>of the email does apply to Debian.
>
>This proposal is in two parts: gems and node packages. The general issue
>applies to both, the project has a need to be able to move and update
>dependencies more quickly given the speed at which platforms like Rails,
>React and Webpack update.
>
>
>Node/NPM Packages
>
>Today, we package nearly all nodejs packages that are required to either
>build the UI or runtime UI dependencies into packages. At current count,
>there are 78 nodejs packages listed in Koji as RPMs. Over the past few
>months, the move to React and updating UI within Foreman core and branching
>into plugins has led to multiple, and at times, compounding packaging and
>thus nightly breakages. One of those breakages lasted over a week with
>multiple developers spending many hours tracking down the issue. The speed
>at which these changes are made makes keeping up successful nightly builds
>difficult. The git repository gets well ahead of the packaging which leads
>to compounding issues. This has also been the cause of stable version
>release delays.
>
>Proposal:
> Deprecate nodejs packages in favor of foreman-assets or a new RPM
>foreman-node-modules that contains a source tarball of node_modules/
>packaged into a simple RPM that is used as a build dependency. This tarball
>would be regenerated, and the package bumped, as dependency updates are
>needed.

I'm not sure how wise this is because I'm not that familiar with NPM and
webpack. After some experimenting I've grown to strongly dislike the
entire JS ecosystem. Their ideas about package management differ
strongly from mine.

I do wonder how we'll handle plugins which have their own dependencies.
Will they ship a foreman-PLUGIN-assets with their own (bundled)
dependencies? What if there's a conflict? How will we monitor security
issues? All the traditional issues that bundling brings along.

In short: I'd like to avoid this but wonder if we'll be forced to
because of their policies.

One side note: it looks like our RPMs are growing a LOT by this and
cause our mirrors to need a lot more disk space and bandwidth. While
perhaps cheap, it is something to consider. node_modules is big.

>Ruby Gems
>
>Today we package all required rubygems as RPMs and utilize a thirdparty
>Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack
>(sclo-ror42). The team that manages the RHSCLs has already released Ruby
>2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no
>plans to release any further SCLs for the Rails stack. This means, in
>addition to our own dependencies, we need to build and manage the Rails
>stack for the versions we want. This adds more packaging burden on the RPM
>side. GIven this, we have a few options:
>
> 1) Build and manage our own SCLs for every version of Rails from here on
>out
> 2) Vendorize Rails and thirdparty gems into a one or more base RPMs
>(e.g. an RPM for rails stack and one for thirdparty dependencies)
>
>Option 2, to vendorize, is a deviation from our prior practices in the area
>of production deployment. Thus, I am reaching out to the community to get
>feedback. One interesting consideration for vendorizing is when containers
>are considered and having the ability to build them using 'bundle install'
>versus using RPM based installation.
>
>In theory vendorizing allows the community to move faster, keep up to date
>with dependencies more easily and reduces the burden on RPM packaging to
>keep nightly builds flowing. However, this does stray from the standard RPM
>based installation benefits typically associated to it.

I'd strongly prefer 1) because I believe it leads to a higher quality.
While it does have downsides, presently I don't think it's that much of
an issue. I'm also willing to invest more time here to ensure it'll
become smoother.

··· On Sat, Oct 14, 2017 at 02:24:58PM -0400, Eric D Helms wrote:
6

Hi,

··· On Mon, Oct 16, 2017 at 02:36:27PM +0100, Greg Sutcliffe wrote: > > Ruby Gems > > > > Option 2, to vendorize, is a deviation from our prior practices in > > the area of production deployment. Thus, I am reaching out to the > > community to get feedback. One interesting consideration for > > vendorizing is when containers are considered and having the ability > > to build them using 'bundle install' versus using RPM based > > installation. > > Vendoring hasn't (to my knowledge) caused many issues for Debian users > (Michael?), and having consistent build processes makes it easier for > anyone to support users on different OSs.

From a user’s perspective it’s not that bad, but IMHO the current
approach does have some downsides regarding plugins and I’m unsure if a
big plugin like Katello can be handled by the current DEB way of doing
things.

Regards

Michael Moll

7

I would suggest the decision be made based on the assumption that the
foreman community owns all the work for options 1 and 2. I have no
special instights, but I would hate to assume we could package based on
another communities work. Assume this commuity owns it all, which way is
better/easier/quicker/less error prone.

– bk

··· On 10/16/2017 04:36 AM, Michael Moll wrote: > I don't really have stakes in RPMs, but I'd try to stick with option 1 > and I even expect that there's some demand for Rails SCLs at least in > the RHEL/CentOS world by other projects, too. > > Regards
8

I spoke to a few people I know about this, and it seems there's not
much appetite for making new SCLs. We might be able to attract
contributors once it's created, but I think we should assume the effort
for creating/maintaining SCLs will fall on us initially.

Do we have any conclusions on this thread? It's going to matter for
1.17 which is getting closer by the day.

Greg

··· On Mon, 2017-10-16 at 14:36 +0100, Greg Sutcliffe wrote: > > That said, I've not really been involved with the RPMs, so I'm unsure > if this causes a bigger headache for Yum users than Apt users. I'm > also unsure of the work required to create an SCL, but if it's non- > trivial then I'd be looking to CentOS to collaborate on a Rails SCL > for everyone to use - for just ourselves, then vendoring seems > easier.
9

Thanks for bringing up the discussion!

As one who has been bitten multiple times by the issue of building RPMs for
node modules, a definite +1 (or more like +1000) from me to vendoring the
entire node_modules directory.
This will save a lot of time and effort on the part of maintainers that
will no longer have to wrestle with koji for every module, as well as
reduce the load on the builders since we won't be wasting cycles building
RPMs for the sole purpose of consuming them during our assets compilation.

However, I'm not sure if this approach should also apply to ruby gems. In
general, the number of gems and the frequency in which they are updates are
both lower, and unlike the node RPMs they are actually used during runtime
and not only during build time, making sense for them to be separate RPMs
that can be in some cases even be upgraded without upgrading the core
packages (which again doesn't apply to node modules which will require a
recompile anyways).

··· On Mon, Oct 16, 2017 at 4:50 PM, Michael Moll wrote:

Hi,

On Mon, Oct 16, 2017 at 02:36:27PM +0100, Greg Sutcliffe wrote:

Ruby Gems

Option 2, to vendorize, is a deviation from our prior practices in
the area of production deployment. Thus, I am reaching out to the
community to get feedback. One interesting consideration for
vendorizing is when containers are considered and having the ability
to build them using ‘bundle install’ versus using RPM based
installation.

Vendoring hasn’t (to my knowledge) caused many issues for Debian users
(Michael?), and having consistent build processes makes it easier for
anyone to support users on different OSs.

From a user’s perspective it’s not that bad, but IMHO the current
approach does have some downsides regarding plugins and I’m unsure if a
big plugin like Katello can be handled by the current DEB way of doing
things.

Regards

Michael Moll

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Have a nice day,
Tomer Brisker
Red Hat Engineering

10

> Today we package all required rubygems as RPMs and utilize a thirdparty
> Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack
> (sclo-ror42). The team that manages the RHSCLs has already released Ruby
> 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no
> plans to release any further SCLs for the Rails stack. This means, in
> addition to our own dependencies, we need to build and manage the Rails
> stack for the versions we want. This adds more packaging burden on the RPM
> side. GIven this, we have a few options:
>
> 1) Build and manage our own SCLs for every version of Rails from here
on
> out
> 2) Vendorize Rails and thirdparty gems into a one or more base RPMs
> (e.g. an RPM for rails stack and one for thirdparty dependencies)

Not sure if this has already been considered: use bundler to package the
app. In this case all dependencies, including rails will be cached locally,
and we could then wrap the result into an rpm. Not sure how plugins would
fit into this though.

-d

··· On Tue, Oct 17, 2017 at 11:17 AM, Bryan Kearney wrote:

On 10/16/2017 04:36 AM, Michael Moll wrote:

I don’t really have stakes in RPMs, but I’d try to stick with option 1
and I even expect that there’s some demand for Rails SCLs at least in
the RHEL/CentOS world by other projects, too.

Regards

I would suggest the decision be made based on the assumption that the
foreman community owns all the work for options 1 and 2. I have no
special instights, but I would hate to assume we could package based on
another communities work. Assume this commuity owns it all, which way is
better/easier/quicker/less error prone.

– bk

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

11

Personally I feel an updated RoR SCL is the way to go for 1.17 and to
prove that I'm willing to invest the time to make it a reality. After
1.16 RC2 is out I'm going to spend time on it.

··· On Mon, Oct 23, 2017 at 09:52:39AM +0100, Greg Sutcliffe wrote: >On Mon, 2017-10-16 at 14:36 +0100, Greg Sutcliffe wrote: >> >> That said, I've not really been involved with the RPMs, so I'm unsure >> if this causes a bigger headache for Yum users than Apt users. I'm >> also unsure of the work required to create an SCL, but if it's non- >> trivial then I'd be looking to CentOS to collaborate on a Rails SCL >> for everyone to use - for just ourselves, then vendoring seems >> easier. > >I spoke to a few people I know about this, and it seems there's not >much appetite for making new SCLs. We might be able to attract >contributors once it's created, but I think we should assume the effort >for creating/maintaining SCLs will fall on us initially. > >Do we have any conclusions on this thread? It's going to matter for >1.17 which is getting closer by the day.
12

I agree with Tomer here, ditto.

On the other hand, the way how rubygems are being packaged makes
package patches (RPM patch file sets) really hard (in short - it's a
hack). Since I also touch packaging work here and there, I know how
much work is to maintain full stack like Ruby on Rails. So I can
imagine having RoR vendored completely, if and only if this goes into
downstream without much changes (I see this as possible problem).
There are advantages of this approach, we could easily ship more gems,
e.g. development gems, without much effort. Every gem vendored would
need to be reviewed (changelog and license) for sure, just like
others.

So I give Node vendoring +1 and both B1 and B2 proposals also +1 +1.

LZ

··· On Tue, Oct 17, 2017 at 9:56 AM, Tomer Brisker wrote: > Thanks for bringing up the discussion! > > As one who has been bitten multiple times by the issue of building RPMs for > node modules, a definite +1 (or more like +1000) from me to vendoring the > entire node_modules directory. > This will save a lot of time and effort on the part of maintainers that will > no longer have to wrestle with koji for every module, as well as reduce the > load on the builders since we won't be wasting cycles building RPMs for the > sole purpose of consuming them during our assets compilation. > > However, I'm not sure if this approach should also apply to ruby gems. In > general, the number of gems and the frequency in which they are updates are > both lower, and unlike the node RPMs they are actually used during runtime > and not only during build time, making sense for them to be separate RPMs > that can be in some cases even be upgraded without upgrading the core > packages (which again doesn't apply to node modules which will require a > recompile anyways). > > On Mon, Oct 16, 2017 at 4:50 PM, Michael Moll wrote: >> >> Hi, >> >> On Mon, Oct 16, 2017 at 02:36:27PM +0100, Greg Sutcliffe wrote: >> > > Ruby Gems >> > > >> > > Option 2, to vendorize, is a deviation from our prior practices in >> > > the area of production deployment. Thus, I am reaching out to the >> > > community to get feedback. One interesting consideration for >> > > vendorizing is when containers are considered and having the ability >> > > to build them using 'bundle install' versus using RPM based >> > > installation. >> > >> > Vendoring hasn't (to my knowledge) caused many issues for Debian users >> > (Michael?), and having consistent build processes makes it easier for >> > anyone to support users on different OSs. >> >> From a user's perspective it's not that bad, but IMHO the current >> approach does have some downsides regarding plugins and I'm unsure if a >> big plugin like Katello can be handled by the current DEB way of doing >> things. >> >> Regards >> -- >> Michael Moll >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Have a nice day, > Tomer Brisker > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

13

Plugins do present the most complicated part of this process given they
would each potentially be requiring a few independent gems separate from
the core stack and we would not want bloat. Seems we'd need either:

  1. a single gem stack that plugins contribute to and is renegenrated with
    all them enabled
  2. a method for plugins to generate the bundle install, and pick out just
    the gems they need to contribute on top of the core stack
··· On Tue, Oct 17, 2017 at 6:42 PM, Dmitri Dolguikh wrote:

Today we package all required rubygems as RPMs and utilize a thirdparty
Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack
(sclo-ror42). The team that manages the RHSCLs has already released Ruby
2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no
plans to release any further SCLs for the Rails stack. This means, in
addition to our own dependencies, we need to build and manage the Rails
stack for the versions we want. This adds more packaging burden on the
RPM
side. GIven this, we have a few options:

  1. Build and manage our own SCLs for every version of Rails from here
    on
    out
  2. Vendorize Rails and thirdparty gems into a one or more base RPMs
    (e.g. an RPM for rails stack and one for thirdparty dependencies)

Not sure if this has already been considered: use bundler to package the
app. In this case all dependencies, including rails will be cached locally,
and we could then wrap the result into an rpm. Not sure how plugins would
fit into this though.

-d

On Tue, Oct 17, 2017 at 11:17 AM, Bryan Kearney bryan.kearney@gmail.com > wrote:

On 10/16/2017 04:36 AM, Michael Moll wrote:

I don’t really have stakes in RPMs, but I’d try to stick with option 1
and I even expect that there’s some demand for Rails SCLs at least in
the RHEL/CentOS world by other projects, too.

Regards

I would suggest the decision be made based on the assumption that the
foreman community owns all the work for options 1 and 2. I have no
special instights, but I would hate to assume we could package based on
another communities work. Assume this commuity owns it all, which way is
better/easier/quicker/less error prone.

– bk

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

14

hi,

··· On Tue, Oct 17, 2017 at 03:42:50PM -0700, Dmitri Dolguikh wrote: > > 2) Vendorize Rails and thirdparty gems into a one or more base RPMs > > (e.g. an RPM for rails stack and one for thirdparty dependencies) > > Not sure if this has already been considered: use bundler to package the > app. In this case all dependencies, including rails will be cached locally, > and we could then wrap the result into an rpm. Not sure how plugins would > fit into this though.

I’m not 100% sure if you meant it like that, but basically this is what
we’re doing in the current DEB approach. The “foreman” package is the
result of “bundle package”, the subpackages drop in a file or two each
into bundler.d/ and a plugin package is its gem (and if needed,
dependent gems) to add into the current bundle and precompiled assets.

Regards

Michael Moll

15

The conversation has been open for 2 weeks now, and I appreciate all of the
feedback and conversation. I am going to summarize the conversations as
they stand and outline what I believe next steps are based on the feedback.

RPMs

I performed a quick tally of those that responded and essentially got the
following counts on SCL vs. vendor.

SCL: 4
Vendorizing: 2

Further, there appeared to be a lot of unanswered technical questions
around how we maintain a vendorized stack with plugins that need to add
dependencies. Based on this feedback, I believe the goal should be to
create a new Rails SCL that we own and maintain. As for the plan, the how,
of creating and maintaining this new SCL, I will start a new thread to
discuss the plan and improvements we can make along the way.

NPM

Across the board, everyone appeared to be in favor of vendorizing our NPM
modules to reduce the frequency of breakages and to allow the UI work that
is on going across Foreman and plugins to continue at a rapid pace. I'll
start a similar thread to outline and discuss the changes for this.

Eric

··· On Sun, Oct 29, 2017 at 5:43 PM, Ewoud Kohl van Wijngaarden < ewoud@kohlvanwijngaarden.nl> wrote:

On Mon, Oct 23, 2017 at 09:52:39AM +0100, Greg Sutcliffe wrote:

On Mon, 2017-10-16 at 14:36 +0100, Greg Sutcliffe wrote:

That said, I’ve not really been involved with the RPMs, so I’m unsure
if this causes a bigger headache for Yum users than Apt users. I’m
also unsure of the work required to create an SCL, but if it’s non-
trivial then I’d be looking to CentOS to collaborate on a Rails SCL
for everyone to use - for just ourselves, then vendoring seems
easier.

I spoke to a few people I know about this, and it seems there’s not
much appetite for making new SCLs. We might be able to attract
contributors once it’s created, but I think we should assume the effort
for creating/maintaining SCLs will fall on us initially.

Do we have any conclusions on this thread? It’s going to matter for
1.17 which is getting closer by the day.

Personally I feel an updated RoR SCL is the way to go for 1.17 and to
prove that I’m willing to invest the time to make it a reality. After 1.16
RC2 is out I’m going to spend time on it.

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

16

> Plugins do present the most complicated part of this process given they
> would each potentially be requiring a few independent gems separate from the
> core stack and we would not want bloat. Seems we'd need either:
>
> 1) a single gem stack that plugins contribute to and is renegenrated with
> all them enabled
> 2) a method for plugins to generate the bundle install, and pick out just
> the gems they need to contribute on top of the core stack

The disadvantage of 2) comes, when there are two plugins, that both need
the same dependency, that is not in the core and using both plugins can
lead to version collisions (we've seen that in the deb packaging every
now and then).
Although, having multiple versions of gems usually is not that much of
a big deal, as long as they are compatible with the plugins, it's
seems a bit weird
that the specific versions being used would depend on the specific plugins
being installed.

The disadvantage of the the single plugins gem stack is, that update of
any dependency there means that the end user needs to re-download the
whole bundle
again. This is probably not issue for most of the dependencies, that
we don't update that
often (the part of the stack). But there are gems, such as Dynflow,
that are more
related to the code and it would be beneficial to have an ability to
deliver new versions
asynchronously. Thinking about it a bit more, we usually need the fast
updates in
nightlies, where it's probably not much of a deal for the end user, and having
the gem stack rpm updated with every upstream release seems like it could work.

There would be still possibility for any plugin, to bring in their own
dependency,
that would not be in the common stack, if needed, but the recommended
would be to get into the Foreman ecosystem to get the deps into the
common stack.

So for now, 1) is winning for me. This might be also an entry-point for testing
the plugins together.

Another potential move could be tackling how the
plugins are getting enable in the first place. If the bits and pieces
for all the plugins
would be available every time, than the act of installing a plugin would
be just about enabling loading the corresponding gem (as opposed for
need to installing
it first). It would make it also easier to make the plugins
discoverable: there is of course
more work to be done for making that working, but just exploring what
it might mean.

It's also good to thing about how that could influence the
containerization work. We could
still build the containers at the end-user side, with a subset of
plugins installed, but
with the all-plugins bundle, we could distribute the images themselves.

There would be additional requirement on the plugin developers to sync
even more with
the core releases, but I think it's a good thing.

– Ivan

··· On Wed, Oct 18, 2017 at 1:46 AM, Eric D Helms wrote:

On Tue, Oct 17, 2017 at 6:42 PM, Dmitri Dolguikh witlessbird@gmail.com > wrote:

Today we package all required rubygems as RPMs and utilize a thirdparty
Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack
(sclo-ror42). The team that manages the RHSCLs has already released Ruby
2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no
plans to release any further SCLs for the Rails stack. This means, in
addition to our own dependencies, we need to build and manage the Rails
stack for the versions we want. This adds more packaging burden on the
RPM
side. GIven this, we have a few options:

  1. Build and manage our own SCLs for every version of Rails from here
    on
    out
  2. Vendorize Rails and thirdparty gems into a one or more base RPMs
    (e.g. an RPM for rails stack and one for thirdparty dependencies)

Not sure if this has already been considered: use bundler to package the
app. In this case all dependencies, including rails will be cached locally,
and we could then wrap the result into an rpm. Not sure how plugins would
fit into this though.

-d

On Tue, Oct 17, 2017 at 11:17 AM, Bryan Kearney bryan.kearney@gmail.com >> wrote:

On 10/16/2017 04:36 AM, Michael Moll wrote:

I don’t really have stakes in RPMs, but I’d try to stick with option 1
and I even expect that there’s some demand for Rails SCLs at least in
the RHEL/CentOS world by other projects, too.

Regards

I would suggest the decision be made based on the assumption that the
foreman community owns all the work for options 1 and 2. I have no
special instights, but I would hate to assume we could package based on
another communities work. Assume this commuity owns it all, which way is
better/easier/quicker/less error prone.

– bk

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

17

> hi,
>
> > > 2) Vendorize Rails and thirdparty gems into a one or more base RPMs
> > > (e.g. an RPM for rails stack and one for thirdparty dependencies)
> >
> > Not sure if this has already been considered: use bundler to package the
> > app. In this case all dependencies, including rails will be cached
> locally,
> > and we could then wrap the result into an rpm. Not sure how plugins would
> > fit into this though.
>
> I'm not 100% sure if you meant it like that, but basically this is what
> we're doing in the current DEB approach. The "foreman" package is the
> result of "bundle package", the subpackages drop in a file or two each
> into bundler.d/ and a plugin package is its gem (and if needed,
> dependent gems) to add into the current bundle and precompiled assets.
>

How do the plugins isolate their dependent gems? i.e. is it a script or
build tool or just knowledge to grab the right gems and pop them in?

··· On Wed, Oct 18, 2017 at 3:01 PM, Michael Moll wrote: > On Tue, Oct 17, 2017 at 03:42:50PM -0700, Dmitri Dolguikh wrote:

Regards

Michael Moll

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

18

There's two forms. For gems only used by the plugin, we add them to the
package as a list of deps - see [1] for an example.

Where more than one gem needs a dep, we move it to a separate package
that can be depended on, e.g. deface [2,3]

Greg

[1] https://github.com/theforeman/foreman-packaging/blob/deb/develop/pl
ugins/ruby-foreman-graphite/debian/gem.list
[2] https://github.com/theforeman/foreman-packaging/tree/deb/develop/pl
ugins/ruby-foreman-deface
[3] https://github.com/theforeman/foreman-packaging/blob/deb/develop/pl
ugins/ruby-foreman-column-view/debian/control#L11

··· On Wed, 2017-10-18 at 15:53 -0400, Eric D Helms wrote: > > How do the plugins isolate their dependent gems? i.e. is it a script > or build tool or just knowledge to grab the right gems and pop them > in?