Katello Pipeline Failure: Foreman Tasks, Templates and Sinatra

All,

The current Katello pipeline failure has proven elusive and after a few
hours of digging I have arrived at the cause, but not at a solution. This
where I need some help, specifically from Marek and Ivan who have worked in
these areas the most. I will try to lay out below the case for what I
believe is the issue in as much detail as I can.

The story begins with the following error during database seeding (stack
trace incoming):

Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb
rake aborted!
wrong number of arguments (0 for 1)
/opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1039:in
template' /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1602:inblock (2 levels) in delegate'
/opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:143:in
attribute_change' /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:inblock in changes'
/opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in
map' /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:inchanges'
/usr/share/foreman/app/models/concerns/strip_whitespace.rb:9:in
strip_spaces' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:407:inrun__4288197166916692034__validation_
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in
__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in_run_validation_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in
run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations/callbacks.rb:53:inrun_validations!'
/opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations.rb:194:in
valid?' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:69:invalid?'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:77:in
perform_validations' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:insave'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in
save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:inblock (2 levels) in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in
block in with_transaction_returning_st /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:intr
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:inwith_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in
block in save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:inrollback_active_record_state!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in
save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:45:increate'
/usr/share/foreman/db/seeds.d/07-provisioning_templates.rb:72:in `block (2
levels) in <top (required)>'

The stack trace points at that:

  1. The provisioning template seed is the start of the issue, that line is,
    specifically where creation occurs
  2. Call is made to strip_whitespace in Foreman that runs 'self.changes'
  3. The call fails in a Sinatra base class

This issue is specific to Katello, or, to be more precise, Foreman Tasks
since that is where Sinatra is used to host the Dynflow Webconsole [1].
When the Webconsole is mounted, an instance of Sinatra is instantiated [2].
When Sinatra is required and instantiated the Sinatra main file is included
which if you look at [3] also includes, at the top level of all things, the
Sinatra::Delegator class (line 28). If you follow the trail to the
Delegator [4], you will see it adds a delegate method and declares below it
[5] what methods to delegate on. And this where provisioning templates
enter the mix, because, with the new addition of the Template class and the
template attribute [6] when the creation occurs, and 'template' is called
it is delegated to the Sinatra template method [7] which takes 1 argument
and gets us back to the original stacktrace.

I don't know the right solution, or how to predict errors like this but I
can conjecture that at least:

  1. Emphasizing code isolation and the use of namespacing can help prevent
    global stack pollution
  2. This is a great reason for us to double down on a full stack (foreman +
    supported plugins) solution

Ideas welcome to so that we can get the pipeline flowing again.

Eric

[1]


[2]

[3] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/main.rb
[4] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1597
[5] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1608
[6]
https://github.com/theforeman/foreman/blob/develop/app/models/template.rb#L4
[7] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1039

Thanks for looking into that: I was not aware about this global addition when requiring sinatra. I will try switching that to `require "sinatra/base" instead.

– Ivan

··· ----- Original Message ----- > All, > > The current Katello pipeline failure has proven elusive and after a few > hours of digging I have arrived at the cause, but not at a solution. This > where I need some help, specifically from Marek and Ivan who have worked in > these areas the most. I will try to lay out below the case for what I > believe is the issue in as much detail as I can. > > The story begins with the following error during database seeding (stack > trace incoming): > > Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb > rake aborted! > wrong number of arguments (0 for 1) > /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1039:in > `template' > /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1602:in > `block (2 levels) in delegate' > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:143:in > `attribute_change' > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in > `block in changes' > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in > `map' > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in > `changes' > /usr/share/foreman/app/models/concerns/strip_whitespace.rb:9:in > `strip_spaces' > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:407:in > `_run__4288197166916692034__validation__ > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in > `__run_callback' > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in > `_run_validation_callbacks' > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in > `run_callbacks' > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations/callbacks.rb:53:in > `run_validations!' > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations.rb:194:in > `valid?' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:69:in > `valid?' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:77:in > `perform_validations' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in > `save' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in > `save' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in > `block (2 levels) in save' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in > `block in with_transaction_returning_st > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in > `tr > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in > `transaction' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in > `with_transaction_returning_status' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in > `block in save' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in > `rollback_active_record_state!' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in > `save' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:45:in > `create' > /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb:72:in `block (2 > levels) in ' > > > The stack trace points at that: > > 1) The provisioning template seed is the start of the issue, that line is, > specifically where creation occurs > 2) Call is made to strip_whitespace in Foreman that runs 'self.changes' > 3) The call fails in a Sinatra base class > > This issue is specific to Katello, or, to be more precise, Foreman Tasks > since that is where Sinatra is used to host the Dynflow Webconsole [1]. > When the Webconsole is mounted, an instance of Sinatra is instantiated [2]. > When Sinatra is required and instantiated the Sinatra main file is included > which if you look at [3] also includes, at the top level of all things, the > Sinatra::Delegator class (line 28). If you follow the trail to the > Delegator [4], you will see it adds a delegate method and declares below it > [5] what methods to delegate on. And this where provisioning templates > enter the mix, because, with the new addition of the Template class and the > template attribute [6] when the creation occurs, and 'template' is called > it is delegated to the Sinatra template method [7] which takes 1 argument > and gets us back to the original stacktrace. > > I don't know the right solution, or how to predict errors like this but I > can conjecture that at least: > > 1) Emphasizing code isolation and the use of namespacing can help prevent > global stack pollution > 2) This is a great reason for us to double down on a full stack (foreman + > supported plugins) solution > > Ideas welcome to so that we can get the pipeline flowing again. > > Eric > > > [1] > https://github.com/theforeman/foreman-tasks/blob/master/config/routes.rb#L22 > [2] > https://github.com/Dynflow/dynflow/blob/edb24af6fe75d821f74fb5c2d48fcf249989d948/lib/dynflow/web_console.rb#L10 > [3] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/main.rb > [4] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1597 > [5] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1608 > [6] > https://github.com/theforeman/foreman/blob/develop/app/models/template.rb#L4 > [7] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1039 > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >

> Thanks for looking into that: I was not aware about this global addition when
> requiring sinatra. I will try switching that to require &quot;sinatra/base&quot; &gt; instead. &gt; &gt; -- Ivan &gt; &gt; &gt; All, &gt; &gt; &gt; &gt; The current Katello pipeline failure has proven elusive and after a few &gt; &gt; hours of digging I have arrived at the cause, but not at a solution. This &gt; &gt; where I need some help, specifically from Marek and Ivan who have worked in &gt; &gt; these areas the most. I will try to lay out below the case for what I &gt; &gt; believe is the issue in as much detail as I can. &gt; &gt; &gt; &gt; The story begins with the following error during database seeding (stack &gt; &gt; trace incoming): &gt; &gt; &gt; &gt; Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb &gt; &gt; rake aborted! &gt; &gt; wrong number of arguments (0 for 1) &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1039:in &gt; &gt;template'
> > /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1602:in
> > block (2 levels) in delegate&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:143:in &gt; &gt;attribute_change'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in
> > block in changes&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in &gt; &gt;map'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in
> > changes&#39; &gt; &gt; /usr/share/foreman/app/models/concerns/strip_whitespace.rb:9:in &gt; &gt;strip_spaces'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:407:in
> > _run__4288197166916692034__validation__ &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in &gt; &gt;__run_callback'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in
> > _run_validation_callbacks&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in &gt; &gt;run_callbacks'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations/callbacks.rb:53:in
> > run_validations!&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations.rb:194:in &gt; &gt;valid?'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:69:in
> > valid?&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:77:in &gt; &gt;perform_validations'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in
> > save&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in &gt; &gt;save'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in
> > block (2 levels) in save&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in &gt; &gt;block in with_transaction_returning_st
> > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in
> > tr &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in &gt; &gt;transaction'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in
> > with_transaction_returning_status&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in &gt; &gt;block in save'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in
> > rollback_active_record_state!&#39; &gt; &gt; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in &gt; &gt;save'
> > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:45:in
> > create&#39; &gt; &gt; /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb:72:inblock (2
> > levels) in <top (required)>'
> >
> >
> > The stack trace points at that:
> >
> > 1) The provisioning template seed is the start of the issue, that line is,
> > specifically where creation occurs
> > 2) Call is made to strip_whitespace in Foreman that runs 'self.changes'
> > 3) The call fails in a Sinatra base class
> >
> > This issue is specific to Katello, or, to be more precise, Foreman Tasks
> > since that is where Sinatra is used to host the Dynflow Webconsole [1].
> > When the Webconsole is mounted, an instance of Sinatra is instantiated [2].
> > When Sinatra is required and instantiated the Sinatra main file is included
> > which if you look at [3] also includes, at the top level of all things, the
> > Sinatra::Delegator class (line 28). If you follow the trail to the
> > Delegator [4], you will see it adds a delegate method and declares below it
> > [5] what methods to delegate on. And this where provisioning templates
> > enter the mix, because, with the new addition of the Template class and the
> > template attribute [6] when the creation occurs, and 'template' is called
> > it is delegated to the Sinatra template method [7] which takes 1 argument
> > and gets us back to the original stacktrace.
> >
> > I don't know the right solution, or how to predict errors like this but I
> > can conjecture that at least:
> >
> > 1) Emphasizing code isolation and the use of namespacing can help prevent
> > global stack pollution
> > 2) This is a great reason for us to double down on a full stack (foreman +
> > supported plugins) solution

At the core or the problem was a bug in sintara 1.3.x that was resolved in
1.4 https://github.com/sinatra/sinatra/commit/46bdb7dcf842afdb3bce57a6753eb4d909b1225d
(you can notice that this is pretty old commit, but it's not part of the rhscl ruby193).

That's also the reason the developers have not noticed the issue until the real production
take in place: before we go any futher, please note that in this case, it's really not about not developing
against the whole stack and definitely was not the case for the templates updates, where the
changes were delivered both to foreman and katello at once.

I've opened a PR against Dynflow to avoid this issue https://github.com/Dynflow/dynflow/pull/154.

What we can however do is making the failure of the nightly pipeline more visible:
I accidentally hit the issue when I wanted to test another Dynflow changes in full stack
production, and the nightlies were already a week failing without anyone noticing.

– Ivan

··· ----- Original Message ----- > ----- Original Message -----

Ideas welcome to so that we can get the pipeline flowing again.

Eric

[1]
https://github.com/theforeman/foreman-tasks/blob/master/config/routes.rb#L22
[2]
https://github.com/Dynflow/dynflow/blob/edb24af6fe75d821f74fb5c2d48fcf249989d948/lib/dynflow/web_console.rb#L10
[3] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/main.rb
[4]
https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1597
[5]
https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1608
[6]
https://github.com/theforeman/foreman/blob/develop/app/models/template.rb#L4
[7]
https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1039


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The new version of dynflow with the fix is released http://koji.katello.org/koji/buildinfo?buildID=22862, let me know if the katello pipeline works correctly now.

– Ivan

··· ----- Original Message ----- > > > ----- Original Message ----- > > Thanks for looking into that: I was not aware about this global addition > > when > > requiring `sinatra`. I will try switching that to `require "sinatra/base" > > instead. > > > > -- Ivan > > > > ----- Original Message ----- > > > All, > > > > > > The current Katello pipeline failure has proven elusive and after a few > > > hours of digging I have arrived at the cause, but not at a solution. This > > > where I need some help, specifically from Marek and Ivan who have worked > > > in > > > these areas the most. I will try to lay out below the case for what I > > > believe is the issue in as much detail as I can. > > > > > > The story begins with the following error during database seeding (stack > > > trace incoming): > > > > > > Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb > > > rake aborted! > > > wrong number of arguments (0 for 1) > > > /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1039:in > > > `template' > > > /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1602:in > > > `block (2 levels) in delegate' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:143:in > > > `attribute_change' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in > > > `block in changes' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in > > > `map' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in > > > `changes' > > > /usr/share/foreman/app/models/concerns/strip_whitespace.rb:9:in > > > `strip_spaces' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:407:in > > > `_run__4288197166916692034__validation__ > > > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in > > > `__run_callback' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in > > > `_run_validation_callbacks' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in > > > `run_callbacks' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations/callbacks.rb:53:in > > > `run_validations!' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations.rb:194:in > > > `valid?' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:69:in > > > `valid?' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:77:in > > > `perform_validations' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in > > > `save' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in > > > `save' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in > > > `block (2 levels) in save' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in > > > `block in with_transaction_returning_st > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in > > > `tr > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in > > > `transaction' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in > > > `with_transaction_returning_status' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in > > > `block in save' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in > > > `rollback_active_record_state!' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in > > > `save' > > > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:45:in > > > `create' > > > /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb:72:in `block > > > (2 > > > levels) in ' > > > > > > > > > The stack trace points at that: > > > > > > 1) The provisioning template seed is the start of the issue, that line > > > is, > > > specifically where creation occurs > > > 2) Call is made to strip_whitespace in Foreman that runs 'self.changes' > > > 3) The call fails in a Sinatra base class > > > > > > This issue is specific to Katello, or, to be more precise, Foreman Tasks > > > since that is where Sinatra is used to host the Dynflow Webconsole [1]. > > > When the Webconsole is mounted, an instance of Sinatra is instantiated > > > [2]. > > > When Sinatra is required and instantiated the Sinatra main file is > > > included > > > which if you look at [3] also includes, at the top level of all things, > > > the > > > Sinatra::Delegator class (line 28). If you follow the trail to the > > > Delegator [4], you will see it adds a delegate method and declares below > > > it > > > [5] what methods to delegate on. And this where provisioning templates > > > enter the mix, because, with the new addition of the Template class and > > > the > > > template attribute [6] when the creation occurs, and 'template' is called > > > it is delegated to the Sinatra template method [7] which takes 1 argument > > > and gets us back to the original stacktrace. > > > > > > I don't know the right solution, or how to predict errors like this but I > > > can conjecture that at least: > > > > > > 1) Emphasizing code isolation and the use of namespacing can help prevent > > > global stack pollution > > > 2) This is a great reason for us to double down on a full stack (foreman > > > + > > > supported plugins) solution > > At the core or the problem was a bug in sintara 1.3.x that was resolved in > 1.4 > https://github.com/sinatra/sinatra/commit/46bdb7dcf842afdb3bce57a6753eb4d909b1225d > (you can notice that this is pretty old commit, but it's not part of the > rhscl ruby193). > > That's also the reason the developers have not noticed the issue until the > real production > take in place: before we go any futher, please note that in this case, it's > really not about not developing > against the whole stack and definitely was not the case for the templates > updates, where the > changes were delivered both to foreman and katello at once. > > I've opened a PR against Dynflow to avoid this issue > https://github.com/Dynflow/dynflow/pull/154. > > What we can however do is making the failure of the nightly pipeline more > visible: > I accidentally hit the issue when I wanted to test another Dynflow changes in > full stack > production, and the nightlies were already a week failing without anyone > noticing. > > -- Ivan > > > > > > > Ideas welcome to so that we can get the pipeline flowing again. > > > > > > Eric > > > > > > > > > [1] > > > https://github.com/theforeman/foreman-tasks/blob/master/config/routes.rb#L22 > > > [2] > > > https://github.com/Dynflow/dynflow/blob/edb24af6fe75d821f74fb5c2d48fcf249989d948/lib/dynflow/web_console.rb#L10 > > > [3] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/main.rb > > > [4] > > > https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1597 > > > [5] > > > https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1608 > > > [6] > > > https://github.com/theforeman/foreman/blob/develop/app/models/template.rb#L4 > > > [7] > > > https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1039 > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "foreman-dev" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to foreman-dev+unsubscribe@googlegroups.com. > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to foreman-dev+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >

> The new version of dynflow with the fix is released
> http://koji.katello.org/koji/buildinfo?buildID=22862, let me know if the
> katello pipeline works correctly now.
>

Awesome, thanks for getting a fix done so quickly! I will re-run the
pipeline and report back either way. Another oddity I noticed because of
this change is that db:seed is what failed, but you wouldn't find that in
the logs of the installer (and it isn't preventing the continuation of the
installer). This increased the debug time since the installer presented the
only error as being the smart proxy failing to register, which produced a
'RecordNotSaved' in the production.log which after digging was because the
'Features' weren't populated which made me think to test db:seed locally.
IIRC db:seed used to fail louder.

> >
> > At the core or the problem was a bug in sintara 1.3.x that was resolved
> in
> > 1.4
> >
> https://github.com/sinatra/sinatra/commit/46bdb7dcf842afdb3bce57a6753eb4d909b1225d
> > (you can notice that this is pretty old commit, but it's not part of the
> > rhscl ruby193).
> >
> > That's also the reason the developers have not noticed the issue until
> the
> > real production
> > take in place: before we go any futher, please note that in this case,
> it's
> > really not about not developing
> > against the whole stack and definitely was not the case for the templates
> > updates, where the
> > changes were delivered both to foreman and katello at once.
>

Agreed. I think it's evidence towards more full stack testing (from a
production, CI point of view) than not developing with more plugins enabled.

> >
> > I've opened a PR against Dynflow to avoid this issue
> > https://github.com/Dynflow/dynflow/pull/154.
> >
> > What we can however do is making the failure of the nightly pipeline more
> > visible:
> > I accidentally hit the issue when I wanted to test another Dynflow
> changes in
> > full stack
> > production, and the nightlies were already a week failing without anyone
> > noticing.
>

Do you have a suggestion for this? Currently it yells in IRC and sends
emails to myself and Justin. Currently, it is configured to do this for
systest so it sends these notifications when installer PRs fail as well. I
could move this all to the 'release_test_katello' job so it only notifies
pipeline failures.

Eric

··· On Fri, Jun 19, 2015 at 5:52 AM, Ivan Necas wrote:

– Ivan

Ideas welcome to so that we can get the pipeline flowing again.

Eric

[1]

https://github.com/theforeman/foreman-tasks/blob/master/config/routes.rb#L22

[2]

https://github.com/Dynflow/dynflow/blob/edb24af6fe75d821f74fb5c2d48fcf249989d948/lib/dynflow/web_console.rb#L10

[3]
https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/main.rb

[4]

https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1597

[5]

https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1608

[6]

https://github.com/theforeman/foreman/blob/develop/app/models/template.rb#L4

[7]

https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1039


You received this message because you are subscribed to the Google
Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it,
send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University

>
> > The new version of dynflow with the fix is released
> > http://koji.katello.org/koji/buildinfo?buildID=22862, let me know if the
> > katello pipeline works correctly now.
> >
>
> Awesome, thanks for getting a fix done so quickly! I will re-run the
> pipeline and report back either way. Another oddity I noticed because of
> this change is that db:seed is what failed, but you wouldn't find that in
> the logs of the installer (and it isn't preventing the continuation of the
> installer). This increased the debug time since the installer presented the
> only error as being the smart proxy failing to register, which produced a
> 'RecordNotSaved' in the production.log which after digging was because the
> 'Features' weren't populated which made me think to test db:seed locally.
> IIRC db:seed used to fail louder.
>
>
> > >
> > > At the core or the problem was a bug in sintara 1.3.x that was resolved
> > in
> > > 1.4
> > >
> > https://github.com/sinatra/sinatra/commit/46bdb7dcf842afdb3bce57a6753eb4d909b1225d
> > > (you can notice that this is pretty old commit, but it's not part of the
> > > rhscl ruby193).
> > >
> > > That's also the reason the developers have not noticed the issue until
> > the
> > > real production
> > > take in place: before we go any futher, please note that in this case,
> > it's
> > > really not about not developing
> > > against the whole stack and definitely was not the case for the templates
> > > updates, where the
> > > changes were delivered both to foreman and katello at once.
> >
>
> Agreed. I think it's evidence towards more full stack testing (from a
> production, CI point of view) than not developing with more plugins enabled.
>
>
> > >
> > > I've opened a PR against Dynflow to avoid this issue
> > > https://github.com/Dynflow/dynflow/pull/154.
> > >
> > > What we can however do is making the failure of the nightly pipeline more
> > > visible:
> > > I accidentally hit the issue when I wanted to test another Dynflow
> > changes in
> > > full stack
> > > production, and the nightlies were already a week failing without anyone
> > > noticing.
> >
>
> Do you have a suggestion for this? Currently it yells in IRC and sends
> emails to myself and Justin. Currently, it is configured to do this for
> systest so it sends these notifications when installer PRs fail as well. I
> could move this all to the 'release_test_katello' job so it only notifies
> pipeline failures.

In general, I'm not a big fan of Jenkins mails on failures to mailling lists, especially
when done for irrelevant things, but I think this might be a good candidate
for such a thing: or at least broaden the amount of people that get
the message (you can add me at least).

– Ivan

··· ----- Original Message ----- > On Fri, Jun 19, 2015 at 5:52 AM, Ivan Necas wrote:

Eric

– Ivan

Ideas welcome to so that we can get the pipeline flowing again.

Eric

[1]

https://github.com/theforeman/foreman-tasks/blob/master/config/routes.rb#L22

[2]

https://github.com/Dynflow/dynflow/blob/edb24af6fe75d821f74fb5c2d48fcf249989d948/lib/dynflow/web_console.rb#L10

[3]
https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/main.rb

[4]

https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1597

[5]

https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1608

[6]

https://github.com/theforeman/foreman/blob/develop/app/models/template.rb#L4

[7]

https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1039


You received this message because you are subscribed to the Google
Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it,
send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I added you to the email list (and moved the email and IRC notification to
only release_test_katello). I tested the new package and seed passed so all
looks good for the pipeline to pass when that package gets pushed out to
the repositories (currently looks like its blocked by the selectjs PR to
the packaging repository).

Eric