Gem requirements

Development
1

After talking with some developers, I thought I'd bring up a discussion about gem requirements in our gem files. It's a problem which we've experienced a number of times in Katello.

To give some background, recently a new version of a gem was released and it broke everything. The problem with these gem release breakages is that it usually holds up development and it tends to shift the burden of finding and fixing the problem to people who are not familiar with the gem.

Currently, we have several gem dependencies in our gem files that have no requirements at all. This means that we could be using any version from 0.0.1 to infinity. Moreover, it means that people could be developing against different versions of the gem and we could be using different versions of the gem in development then what's being packaged and released.

I know that we have to support multiple platforms and thus multiple versions of a gem but couldn't we lock down our gem requirements to a range that includes those versions? It seems like we should be able to check for new gem versions before releases, and then package them and update our gem files. This way we could test them in a controlled environment instead of being blindsided by new versions being released into the wild.

Optimally, I'd like to be as strict as possible but I think at the very least we could lock down gems to their minor version numbers (e.g. "~> 0.2" or "< 1.0.0", ">= 0.2.1") as major versions almost inevitably bring API changes and thus breakages.

I also realize that there's a caveat when it comes to Katello: we require gems that Foreman also supplies. I think it's fine not to have any requirements for those gems. I'm mostly speaking to the gems that Foreman doesn't require.

Any thoughts?

David

2

As one of the aforementioned developers I wholeheartedly agree.

The only benefit I can see of not locking down a max version is that we may be unaware of a security fix in a gem and get it "for free".

Has that ever happened? And, if so, how many times compared to the number of times the build has broken because of our lack of a max version?

Cheers,
Walden

··· ----- Original Message ----- From: "David Davis" To: "foreman-dev" Sent: Thursday, May 1, 2014 2:00:41 PM Subject: [foreman-dev] Gem requirements

After talking with some developers, I thought I’d bring up a discussion about gem requirements in our gem files. It’s a problem which we’ve experienced a number of times in Katello.

To give some background, recently a new version of a gem was released and it broke everything. The problem with these gem release breakages is that it usually holds up development and it tends to shift the burden of finding and fixing the problem to people who are not familiar with the gem.

Currently, we have several gem dependencies in our gem files that have no requirements at all. This means that we could be using any version from 0.0.1 to infinity. Moreover, it means that people could be developing against different versions of the gem and we could be using different versions of the gem in development then what’s being packaged and released.

I know that we have to support multiple platforms and thus multiple versions of a gem but couldn’t we lock down our gem requirements to a range that includes those versions? It seems like we should be able to check for new gem versions before releases, and then package them and update our gem files. This way we could test them in a controlled environment instead of being blindsided by new versions being released into the wild.

Optimally, I’d like to be as strict as possible but I think at the very least we could lock down gems to their minor version numbers (e.g. “~> 0.2” or “< 1.0.0”, “>= 0.2.1”) as major versions almost inevitably bring API changes and thus breakages.

I also realize that there’s a caveat when it comes to Katello: we require gems that Foreman also supplies. I think it’s fine not to have any requirements for those gems. I’m mostly speaking to the gems that Foreman doesn’t require.

Any thoughts?

David

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

That's the purpose of Jenkins - to tell us things are broken. If we
never had broken gems, then the test_develop job would be redundant,
since every PR is code-tested by test_pull_request. Jenkins gives us a
fast heads-up that although all code is tested, something has changed
in our dependencies. In my experience, knowing that it's almost always
a gem which breaks things means you can grab a list of recently
updated gems from Rubygems, and figure out which one was the problem,
in a reasonable timeframe.

Given Ivan's points, and that Jenkins will tell us about broken gems
very quicky, I think I'll have to agree with Ivan. Knowing we have
work to do to update a gem is preferable to still being on 0.0.5 when
3.7 is available…(alert: made-up numbers :P)

··· On 1 May 2014 19:00, David Davis wrote: > To give some background, recently a new version of a gem was released and it broke everything.
4

> Any thoughts?

But isn't this Bundler's job? On development environments, we use
Bundler, so make sure your lock fine is in git and you are fine.

On production we do rely on RPM dependencies which should only be
upgraded when we test things. If this is your case, we should improve
the review process of getting new builds into the repository.

I don't think any versioning mechanism will help with this issue, it
will only postpone the breakage to the point when we will "think" that
we are safe to rollout major version change. Boom, then it will blow up.

We need to change the process, not versioning scheme. If I understand
your words correctly. The right point to catch those things is before
someone submits a build into koji. You refer to thirdparty gems, that
means before someone submits the build from our thirdparty repository.
Maybe we can create a hook to warn folks before submitting bigger
changes, but the question is - how do we test that the new version does
not break plugins?

And I think the answer is Mr. Jenkins.

One observation - when we build new dependency and it goes to nightly
repos, it is still quite easy to fallback. Untagging a build is just a
single koji command as well as regenerating and syncing the repo. So
recovery is easy, maybe it would be enough to test against nightly repos
three times a day instead of testing each individual thirdparty build
(against all the plugins). Could be painfully slow.

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

5

As much as unpleasant it is, doing this by default is not the best thing to do IMO,
as we can easily loose touch with the upstream changes of that gems. Of course
it will cause some slow down while doing some stuff, but knowing that we're not
compatible with the next version (ideally with comments why the gem version is lockes on some version range).

-1 from me

– Ivan

··· ----- Original Message ----- > As one of the aforementioned developers I wholeheartedly agree. > > The only benefit I can see of not locking down a max version is that we may > be unaware of a security fix in a gem and get it "for free". > > Has that ever happened? And, if so, how many times compared to the number of > times the build has broken because of our lack of a max version? > > Cheers, > Walden > > ----- Original Message ----- > From: "David Davis" > To: "foreman-dev" > Sent: Thursday, May 1, 2014 2:00:41 PM > Subject: [foreman-dev] Gem requirements > > After talking with some developers, I thought I'd bring up a discussion about > gem requirements in our gem files. It's a problem which we've experienced a > number of times in Katello. > > To give some background, recently a new version of a gem was released and it > broke everything. The problem with these gem release breakages is that it > usually holds up development and it tends to shift the burden of finding and > fixing the problem to people who are not familiar with the gem. > > Currently, we have several gem dependencies in our gem files that have no > requirements at all. This means that we could be using any version from > 0.0.1 to infinity. Moreover, it means that people could be developing > against different versions of the gem and we could be using different > versions of the gem in development then what's being packaged and released. > > I know that we have to support multiple platforms and thus multiple versions > of a gem but couldn't we lock down our gem requirements to a range that > includes those versions? It seems like we should be able to check for new > gem versions before releases, and then package them and update our gem > files. This way we could test them in a controlled environment instead of > being blindsided by new versions being released into the wild. > > Optimally, I'd like to be as strict as possible but I think at the very least > we could lock down gems to their minor version numbers (e.g. "~> 0.2" or "< > 1.0.0", ">= 0.2.1") as major versions almost inevitably bring API changes > and thus breakages. > > I also realize that there's a caveat when it comes to Katello: we require > gems that Foreman also supplies. I think it's fine not to have any > requirements for those gems. I'm mostly speaking to the gems that Foreman > doesn't require. > > Any thoughts? > > David > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >
6

> As much as unpleasant it is, doing this by default is not the best thing to
> do IMO,
> as we can easily loose touch with the upstream changes of that gems. Of
> course
> it will cause some slow down while doing some stuff, but knowing that we're
> not
> compatible with the next version (ideally with comments why the gem version
> is lockes on some version range).
>
> -1 from me

Most of our gems are already locked down. We have to track upstream changes for them somehow so why not just do them all the same way?

> That's the purpose of Jenkins - to tell us things are broken. If we
> never had broken gems, then the test_develop job would be redundant,
> since every PR is code-tested by test_pull_request. Jenkins gives us a
> fast heads-up that although all code is tested, something has changed
> in our dependencies. In my experience, knowing that it's almost always
> a gem which breaks things means you can grab a list of recently
> updated gems from Rubygems, and figure out which one was the problem,
> in a reasonable timeframe.

If my PR breaks in Jenkins though how do I know if it's a gem change or a code change? Also, I've seen the main job break before in Katello when it wasn't gem-related so that's not entirely reliable.

David

7

> > As much as unpleasant it is, doing this by default is not the best thing to
> > do IMO,
> > as we can easily loose touch with the upstream changes of that gems. Of
> > course
> > it will cause some slow down while doing some stuff, but knowing that we're
> > not
> > compatible with the next version (ideally with comments why the gem version
> > is lockes on some version range).
> >
> > -1 from me
>
>
> Most of our gems are already locked down. We have to track upstream changes
> for them somehow so why not just do them all the same way?

Define most:

grep 'dependency' katello.gemspec | wc -l
41
grep 'dependency.*[<~]' katello.gemspec | wc -l
12

grep 'add_dependency' katello.gemspec | wc -
32
grep 'add_dependency.*[<~]' katello.gemspec | wc -l
9

– Ivan

··· ----- Original Message -----

That’s the purpose of Jenkins - to tell us things are broken. If we
never had broken gems, then the test_develop job would be redundant,
since every PR is code-tested by test_pull_request. Jenkins gives us a
fast heads-up that although all code is tested, something has changed
in our dependencies. In my experience, knowing that it’s almost always
a gem which breaks things means you can grab a list of recently
updated gems from Rubygems, and figure out which one was the problem,
in a reasonable timeframe.

If my PR breaks in Jenkins though how do I know if it’s a gem change or a
code change? Also, I’ve seen the main job break before in Katello when it
wasn’t gem-related so that’s not entirely reliable.

David

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

In fact, the last issues with deps is an evidence that it makes sense to locking
the deps, see the issues as they appear upstream and being forced to deal with
them as they appear (also congrats on the nice issue number:):

When people release new version of gem, there are usually quite responsive on
immediate issues and tent to release new version with fixes right away. The longer
the time is, the longer it takes to resolve that (I'm personally guilty of doing that).

··· ----- Original Message ----- > > > ----- Original Message ----- > > > As much as unpleasant it is, doing this by default is not the best thing > > > to > > > do IMO, > > > as we can easily loose touch with the upstream changes of that gems. Of > > > course > > > it will cause some slow down while doing some stuff, but knowing that > > > we're > > > not > > > compatible with the next version (ideally with comments why the gem > > > version > > > is lockes on some version range). > > > > > > -1 from me > > > > > > Most of our gems are already locked down. We have to track upstream changes > > for them somehow so why not just do them all the same way? > > Define `most`: > > grep 'dependency' katello.gemspec | wc -l > 41 > grep 'dependency.*[<~]' katello.gemspec | wc -l > 12 > > grep 'add_dependency' katello.gemspec | wc - > 32 > grep 'add_dependency.*[<~]' katello.gemspec | wc -l > 9 > > -- Ivan > > > > > > > > That's the purpose of Jenkins - to tell us things are broken. If we > > > never had broken gems, then the test_develop job would be redundant, > > > since every PR is code-tested by test_pull_request. Jenkins gives us a > > > fast heads-up that although all code is tested, something has changed > > > in our dependencies. In my experience, knowing that it's almost always > > > a gem which breaks things means you can grab a list of recently > > > updated gems from Rubygems, and figure out which one was the problem, > > > in a reasonable timeframe. > > > > > > If my PR breaks in Jenkins though how do I know if it's a gem change or a > > code change? Also, I've seen the main job break before in Katello when it > > wasn't gem-related so that's not entirely reliable. > > > > David > > > > -- > > You received this message because you are subscribed to the Google Groups > > "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to foreman-dev+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >
9

> In fact, the last issues with deps is an evidence that it makes sense to
> locking

s/locking/not locking/ :slight_smile: is it Friday or what?

··· ----- Original Message -----

the deps, see the issues as they appear upstream and being forced to deal
with
them as they appear (also congrats on the nice issue number:):

https://github.com/pitr/angular-rails-templates/issues/42

When people release new version of gem, there are usually quite responsive on
immediate issues and tent to release new version with fixes right away. The
longer
the time is, the longer it takes to resolve that (I’m personally guilty of
doing that).

----- Original Message -----

----- Original Message -----

As much as unpleasant it is, doing this by default is not the best
thing
to
do IMO,
as we can easily loose touch with the upstream changes of that gems. Of
course
it will cause some slow down while doing some stuff, but knowing that
we’re
not
compatible with the next version (ideally with comments why the gem
version
is lockes on some version range).

-1 from me

Most of our gems are already locked down. We have to track upstream
changes
for them somehow so why not just do them all the same way?

Define most:

grep ‘dependency’ katello.gemspec | wc -l
41
grep ‘dependency.*[<~]’ katello.gemspec | wc -l
12

grep ‘add_dependency’ katello.gemspec | wc -
32
grep ‘add_dependency.*[<~]’ katello.gemspec | wc -l
9

– Ivan

That’s the purpose of Jenkins - to tell us things are broken. If we
never had broken gems, then the test_develop job would be redundant,
since every PR is code-tested by test_pull_request. Jenkins gives us a
fast heads-up that although all code is tested, something has changed
in our dependencies. In my experience, knowing that it’s almost always
a gem which breaks things means you can grab a list of recently
updated gems from Rubygems, and figure out which one was the problem,
in a reasonable timeframe.

If my PR breaks in Jenkins though how do I know if it’s a gem change or a
code change? Also, I’ve seen the main job break before in Katello when it
wasn’t gem-related so that’s not entirely reliable.

David

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

> Define most:
>
> grep 'dependency' katello.gemspec | wc -l
> 41
> grep 'dependency.[<~]' katello.gemspec | wc -l
> 12
>
> grep 'add_dependency' katello.gemspec | wc -
> 32
> grep 'add_dependency.[<~]' katello.gemspec | wc -l

Your search misses some gem requirements like this one:

https://github.com/Katello/katello/blob/master/katello.gemspec#L66

Also, from my original email:

"I also realize that there's a caveat when it comes to Katello: we require gems that Foreman also supplies. I think it's fine not to have any requirements for those gems. I'm mostly speaking to the gems that Foreman doesn't require."

A number of the gems we have (like rails) I think ought not to have a version requirement since their version is being handled by foreman:

https://github.com/theforeman/foreman/blob/develop/Gemfile#L8
https://github.com/Katello/katello/blob/master/katello.gemspec#L25

But it may not be most. It was just a rough guess. And really I was meaning "any gem requirements" by "locked down".

> In fact, the last issues with deps is an evidence that it makes sense to
> locking
> the deps, see the issues as they appear upstream and being forced to deal
> with
> them as they appear (also congrats on the nice issue number:):
>
> https://github.com/pitr/angular-rails-templates/issues/42
>
> When people release new version of gem, there are usually quite responsive on
> immediate issues and tent to release new version with fixes right away. The
> longer
> the time is, the longer it takes to resolve that (I'm personally guilty of
> doing that).

So adding gem requirements doesn't preclude us from getting upstream releases and fixes. From my original email (s/minor/major):

"I think at the very least we could lock down gems to their major version numbers (e.g. "~> 0.2" or "< 1.0.0", ">= 0.2.1") as major versions almost inevitably bring API changes and thus breakages."

Also, if we're worried about security fixes to gems, relying on Jenkins seems like a horrible solution. What if a security release comes out and our Jenkins job doesn't fail? Take for example the gem hooks. We're still on version 0.2.2:

Which was released last year:

https://github.com/apotonick/hooks/releases

My dev environment is at 0.4. What good does that do? Maybe gem requirements isn't the solution but I still wonder: how do we know when to update the RPM packages with the latest release? Obviously, our current solution isn't working.

David

··· ----- Original Message ----- > From: "Ivan Necas" > To: foreman-dev@googlegroups.com > Sent: Friday, May 2, 2014 8:04:39 AM > Subject: Re: [foreman-dev] Gem requirements > > > > ----- Original Message ----- > > > As much as unpleasant it is, doing this by default is not the best thing > > > to > > > do IMO, > > > as we can easily loose touch with the upstream changes of that gems. Of > > > course > > > it will cause some slow down while doing some stuff, but knowing that > > > we're > > > not > > > compatible with the next version (ideally with comments why the gem > > > version > > > is lockes on some version range). > > > > > > -1 from me > > > > > > Most of our gems are already locked down. We have to track upstream changes > > for them somehow so why not just do them all the same way? > > Define `most`: > > grep 'dependency' katello.gemspec | wc -l > 41 > grep 'dependency.*[<~]' katello.gemspec | wc -l > 12 > > grep 'add_dependency' katello.gemspec | wc - > 32 > grep 'add_dependency.*[<~]' katello.gemspec | wc -l > 9 > > -- Ivan > > > > > > > > That's the purpose of Jenkins - to tell us things are broken. If we > > > never had broken gems, then the test_develop job would be redundant, > > > since every PR is code-tested by test_pull_request. Jenkins gives us a > > > fast heads-up that although all code is tested, something has changed > > > in our dependencies. In my experience, knowing that it's almost always > > > a gem which breaks things means you can grab a list of recently > > > updated gems from Rubygems, and figure out which one was the problem, > > > in a reasonable timeframe. > > > > > > If my PR breaks in Jenkins though how do I know if it's a gem change or a > > code change? Also, I've seen the main job break before in Katello when it > > wasn't gem-related so that's not entirely reliable. > > > > David > > > > -- > > You received this message because you are subscribed to the Google Groups > > "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to foreman-dev+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >