Recently, the katello-installer has reached a point on EL6 where it
segfaults every time whenever we attempt to upgrade the puppet-apache
module [1]. This is not a new occurrence as we have had to disable the
inclusion of compute resources in our installer for the same reason [2].
Investigations previously have narrowed this down to the Ruby 1.8.7 garbage
collection. This has led us to have to, in the Apache case, fork Foreman
modules due to their reliance on the newer Apache. This is not a path I
think anyone is excited to go down and will only lead to higher maintenance
costs. This leaves us with 2 options:
A) Port the installer to work on the SCL
B) Find a way to get Ruby 1.8.7 to stop segfaulting on us
Requirements for A
Build the installer as SCL packages (e.g. ruby193-katello-installer)
Build kafo as an SCL package
Build puppet, dependencies and facter as SCL packages
Requirements for B
I honestly do not know what it would take to accomplish this.
Questions
Is there an option I am missing?
What are the pros and cons missing from each option.
I don't envy your situation, I'm very glad it went away for Foreman
installer (#4244) through luck. Our future support for Puppet 4 should
fix this for sure.
However be very careful if you're going to redistribute Puppet built as
an SCL package. When we did it a few years ago (as Puppet was a core
dependency at the time) it was very common to find Puppet masters
accidentally running the SCL rather than non-SCL version, so an
unexpected version of Puppet was running. This caused friction and a
lot of confusion with folks supporting Puppet. Removing our SCL version
of Puppet prevented a lot of misconfigurations.
···
On 06/03/15 02:46, Eric D Helms wrote:
> *Requirements for A*
> *
> *
> * Build the installer as SCL packages (e.g. ruby193-katello-installer)
> * Build kafo as an SCL package
> * Build puppet, dependencies and facter as SCL packages
> Recently, the katello-installer has reached a point on EL6 where it
> segfaults every time whenever we attempt to upgrade the puppet-apache
> module [1]. This is not a new occurrence as we have had to disable the
> inclusion of compute resources in our installer for the same reason [2].
> Investigations previously have narrowed this down to the Ruby 1.8.7 garbage
> collection. This has led us to have to, in the Apache case, fork Foreman
> modules due to their reliance on the newer Apache. This is not a path I
> think anyone is excited to go down and will only lead to higher maintenance
> costs. This leaves us with 2 options:
>
> A) Port the installer to work on the SCL
> B) Find a way to get Ruby 1.8.7 to stop segfaulting on us
>
> Requirements for A
>
> * Build the installer as SCL packages (e.g. ruby193-katello-installer)
> * Build kafo as an SCL package
> * Build puppet, dependencies and facter as SCL packages
>
> Requirements for B
>
> I honestly do not know what it would take to accomplish this.
>
>
> Questions
>
> 1) Is there an option I am missing?
I don't think there's anything practical. Update system ruby (PL did
this on EL5 for 1.8.7), or continuing holding on to pl-apache 1.1.1
are all I can think of.
> 2) What are the pros and cons missing from each option.
> 3) Does anyone have any insight into achieving B?
Anyone know any Ruby core developers? The results of Foreman's
investigations determined it was related to garbage collection, but
the problem seems to have disappeared for Foreman, what did they do?
The foreman catalog is clearly simpler than what's generatd to install
Katello though.
> > Recently, the katello-installer has reached a point on EL6 where it
> > segfaults every time whenever we attempt to upgrade the puppet-apache
> > module [1]. This is not a new occurrence as we have had to disable the
> > inclusion of compute resources in our installer for the same reason [2].
> > Investigations previously have narrowed this down to the Ruby 1.8.7 garbage
> > collection. This has led us to have to, in the Apache case, fork Foreman
> > modules due to their reliance on the newer Apache. This is not a path I
> > think anyone is excited to go down and will only lead to higher maintenance
> > costs. This leaves us with 2 options:
> >
> > A) Port the installer to work on the SCL
> > B) Find a way to get Ruby 1.8.7 to stop segfaulting on us
> >
> > Requirements for A
> >
> > * Build the installer as SCL packages (e.g. ruby193-katello-installer)
> > * Build kafo as an SCL package
> > * Build puppet, dependencies and facter as SCL packages
> >
> > Requirements for B
> >
> > I honestly do not know what it would take to accomplish this.
> >
> >
> > Questions
> >
> > 1) Is there an option I am missing?
>
> I don't think there's anything practical. Update system ruby (PL did
> this on EL5 for 1.8.7), or continuing holding on to pl-apache 1.1.1
> are all I can think of.
>
> > 2) What are the pros and cons missing from each option.
> > 3) Does anyone have any insight into achieving B?
>
> Anyone know any Ruby core developers? The results of Foreman's
> investigations determined it was related to garbage collection, but
> the problem seems to have disappeared for Foreman, what did they do?
> The foreman catalog is clearly simpler than what's generatd to install
> Katello though.
By Foreman, do you mean the foreman installer? If so, I'm guessing is
that we use fewer modules which results in less memory usage. Maybe
advanced Ruby developers can use some profiling tools to see where the
memory goes? pl-apache 1.2.0 started using concat fragments which might
be heavier on memory usage for example.
···
On Fri, Mar 06, 2015 at 09:49:29AM +0100, Stephen Benjamin wrote:
> On Thu, Mar 05, 2015 at 09:46:16PM -0500, Eric D Helms wrote:
> > Anyone know any Ruby core developers? The results of Foreman's
> > investigations determined it was related to garbage collection, but
> > the problem seems to have disappeared for Foreman, what did they do?
> > The foreman catalog is clearly simpler than what's generatd to install
> > Katello though.
>
> By Foreman, do you mean the foreman installer? If so, I'm guessing is
> that we use fewer modules which results in less memory usage. Maybe
> advanced Ruby developers can use some profiling tools to see where the
> memory goes? pl-apache 1.2.0 started using concat fragments which might
> be heavier on memory usage for example.
When I was doing research on this bug last year, I stumbled upon this BZ
where I read this sentence by RWMJones:
"This is a fundamental bug in Ruby 1.8. On this basis, I'm
closing as CANTFIX."
I was able to debug the very same pattern he found coming to the same
conclusion. I was able to reproduce it everytime using GC.stress option
(see the comment 13).
TL;DR - Ruby 1.8.7 is broken by design and will never work again with
some native extensions. The only way out is to use 1.9+ or not to use
any native extensions. I think Puppet hard-depends on few of them.
