Fixing the bundler issues

Development
#1

Hi all,

So, we've been having a discussion for a while now about how to use
bundler and how to fix the problems we have with it. As I understand
it, we have two main problems:

  1. Dealing with optional gems like libvirt, and ensuring at least one
    db gem is installed
  2. Packaging for various OSes

For the former, we can try playing with the --without options. I've
just tested with the mysql gem - this fails to build if the headers
aren't installed. Running "bundle install --path vendor/ --without
mysql" worked fine, even without the headers. So, it might be possible
to write some kind of ./install.sh which examines the system and runs
the correct bundle command. Hacky, but doable.

For packaging, I guess we have two options. One is to use bundler's
own packaging tool, and create one huge package file. Not nice, but
probably simple, and we could easily use that to create a
foreman-sqlite.pkg, foreman-mysql.pkg, etc, which has the right gems
in it.

The alternative is to look at FPM
(https://github.com/jordansissel/fpm) which, in theory, we can use to
spit out RPM and DEB files for each of the gem dependencies, stick
them in the foreman repo, and then require those dependencies in the
main foreman package. Slightly nicer, but slightly more work. we can
also (hopefully) do optional dependencies that way (such as the
libvirt and db gems).

I'm going to have a play with FPM and see if I can get a sample deb of
the second version working for the current develop branch.

Thoughts? Joschi, particularly interested if you want to have a play :smiley:

Greg

#2

> Hi all,
>
> So, we've been having a discussion for a while now about how to use
> bundler and how to fix the problems we have with it. As I understand
> it, we have two main problems:
>
> 1) Dealing with optional gems like libvirt, and ensuring at least one
> db gem is installed
> 2) Packaging for various OSes
>
> For the former, we can try playing with the --without options. I've
> just tested with the mysql gem - this fails to build if the headers
> aren't installed. Running "bundle install --path vendor/ --without
> mysql" worked fine, even without the headers. So, it might be possible
> to write some kind of ./install.sh which examines the system and runs
> the correct bundle command. Hacky, but doable.

FTW you can alter directly the ~foreman/.bundle/config to achieve the same
>
> For packaging, I guess we have two options. One is to use bundler's
> own packaging tool, and create one huge package file. Not nice, but
> probably simple, and we could easily use that to create a
> foreman-sqlite.pkg, foreman-mysql.pkg, etc, which has the right gems
> in it.

that would arch specific package (just mentioning).

>
> The alternative is to look at FPM
> (https://github.com/jordansissel/fpm) which, in theory, we can use to
> spit out RPM and DEB files for each of the gem dependencies, stick
> them in the foreman repo, and then require those dependencies in the
> main foreman package. Slightly nicer, but slightly more work. we can
> also (hopefully) do optional dependencies that way (such as the
> libvirt and db gems).
>
> I'm going to have a play with FPM and see if I can get a sample deb of
> the second version working for the current develop branch.

so does that mean we will be overriding the systems gems (if they
match) to have the same gems regardless of the os version?

e.g. if you have rack gem 1.x in your distro, would we upgrade it?

>
> Thoughts? Joschi, particularly interested if you want to have a play :smiley:

I'm torn between two options.

  1. dump bundler all together in production mode, and use the gemfile
    to generate git submodules (like we used to).
    this would allow us to distribute it all easily, and ensrue we have
    the version that we need… one issue with this approch is the
    complied gems which still needs to be required by the package, and,
    some logic that loads the required classes.
  2. stop using Bundler.setup, and read bundler internal hoping to find
    a better approach.

at the end of the day, bundler was designed to ensure that you get the
same version of gems regardless of where you deploy.
IMHO, that is a conflicting solution with OS packages, and we need to chose.
obviously, my developer side thinks its great, as I only need to test
one version of each component, however my sys admins side think it
sucks.

Ohad

··· On Wed, Mar 7, 2012 at 2:55 PM, Greg Sutcliffe wrote:
#3

The easiest option seems to be to use "bundle package" which (if I
understand right) puts everything into a local cache that can be
distributed. That's a fairly heavy package, and architecture
dependant, but buildable without too much effort on our part.

I'll chat a bit more with the devs here - apparently a lot of people
use and like Bundler, and I can't work out why :slight_smile:

··· On 7 March 2012 13:31, Ohad Levy wrote: > I'm torn between two options. > > 1. dump bundler all together in production mode, and use the gemfile > to generate git submodules (like we used to). > 2. stop using Bundler.setup, and read bundler internal hoping to find > a better approach.
#4

for the developer, bundler is easier then what was there before.
e.g. we just say which version we want, and thats it.

for packaging aware people, this is far from ideal, as it requires a
duplication of the code (e.g. one time puppet on your os, another
time bundled for foreman) etc.

if people don't care, I dont mind distributing bundler (and a bundler
lock file) which freeze the versions of all the gems and distributed
it all within one large rpm (that would probably be the easiest).

this however does mean that, there are no optional components.

Ohad

··· On Wed, Mar 7, 2012 at 9:53 PM, Greg Sutcliffe wrote: > On 7 March 2012 13:31, Ohad Levy wrote: >> I'm torn between two options. >> >> 1. dump bundler all together in production mode, and use the gemfile >> to generate git submodules (like we used to). >> 2. stop using Bundler.setup, and read bundler internal hoping to find >> a better approach. > > The easiest option seems to be to use "bundle package" which (if I > understand right) puts everything into a local cache that can be > distributed. That's a fairly heavy package, and architecture > dependant, but buildable without too much effort on our part. > > I'll chat a bit more with the devs here - apparently a lot of people > use and like Bundler, and I can't work out why :)
#5

> for packaging aware people, this is far from ideal, as it requires a
> duplication of the code (e.g. one time puppet on your os, another
> time bundled for foreman) etc.

Well, if it's exact duplication that's an issue. However, I get the
impression that people like bundler because bugfixes to gems come out
faster than the OS packages can keep up. So is that really dupication?

> if people don't care, I dont mind distributing bundler (and a bundler
> lock file) which freeze the versions of all the gems and distributed
> it all within one large rpm (that would probably be the easiest).
>
> this however does mean that, there are no optional components.

I agree, it feels kind of wrong, but it's starting to feel like we
really need to provide some kind of nightly package again for 0.5. The
optional stuff is an annoyance, but we can possibly work out a build
system for some more granular packages as well as one big fat one.
Maybe we should just do the big fat one to start, and see who
complains :stuck_out_tongue:

Greg

··· On 8 March 2012 08:29, Ohad Levy wrote:
#6

Not to complain, but more to look outside foreman a bit.
If we go the static large rpm route, how will this impact systems where
folks want to have katello and foreman installed on the same system?

Both are ruby, both require bundler, and use puppet etc. Will the two
co-exist peacefully?

Would it be prudent to see how they're handling bundler requirements and
integration related to packaging?

··· On Thu, Mar 8, 2012 at 10:02 AM, Greg Sutcliffe wrote:

I agree, it feels kind of wrong, but it’s starting to feel like we
really need to provide some kind of nightly package again for 0.5. The
optional stuff is an annoyance, but we can possibly work out a build
system for some more granular packages as well as one big fat one.
Maybe we should just do the big fat one to start, and see who
complains :stuck_out_tongue:

Greg

–
During times of universal deceit, telling the truth becomes a revolutionary
act.
George Orwell

#7

>
>
>>
>>
>> I agree, it feels kind of wrong, but it's starting to feel like we
>> really need to provide some kind of nightly package again for 0.5. The
>> optional stuff is an annoyance, but we can possibly work out a build
>> system for some more granular packages as well as one big fat one.
>> Maybe we should just do the big fat one to start, and see who
>> complains :stuck_out_tongue:
>>
>> Greg
>
>
>
> Not to complain, but more to look outside foreman a bit.
> If we go the static large rpm route, how will this impact systems where
> folks want to have katello and foreman installed on the same system?
>
> Both are ruby, both require bundler, and use puppet etc. Will the two
> co-exist peacefully?

They would co exists peacefully, one of the motivation to move to
rails3 is to match the katello stack.

Ohad

··· On Thu, Mar 8, 2012 at 7:42 PM, Jim Perrin wrote: > On Thu, Mar 8, 2012 at 10:02 AM, Greg Sutcliffe > wrote: > > Would it be prudent to see how they're handling bundler requirements and > integration related to packaging? > > -- > During times of universal deceit, telling the truth becomes a revolutionary > act. > George Orwell
#8

>> for packaging aware people, this is far from ideal, as it requires a
>> duplication of the code (e.g. one time puppet on your os, another
>> time bundled for foreman) etc.
>
> Well, if it's exact duplication that's an issue. However, I get the
> impression that people like bundler because bugfixes to gems come out
> faster than the OS packages can keep up. So is that really dupication?
>
>> if people don't care, I dont mind distributing bundler (and a bundler
>> lock file) which freeze the versions of all the gems and distributed
>> it all within one large rpm (that would probably be the easiest).
>>
>> this however does mean that, there are no optional components.
>
> I agree, it feels kind of wrong, but it's starting to feel like we
> really need to provide some kind of nightly package again for 0.5.

I fully agree, if don't have people trying out 0.5, I'm sure there
would be embarrassing bugs in the official release :wink:

··· On Thu, Mar 8, 2012 at 6:02 PM, Greg Sutcliffe wrote: > On 8 March 2012 08:29, Ohad Levy wrote:

The optional stuff is an annoyance, but we can possibly work out a build
system for some more granular packages as well as one big fat one.
Maybe we should just do the big fat one to start, and see who
complains :stuck_out_tongue:

Greg