All,
The current Katello pipeline failure has proven elusive and after a few
hours of digging I have arrived at the cause, but not at a solution. This
where I need some help, specifically from Marek and Ivan who have worked in
these areas the most. I will try to lay out below the case for what I
believe is the issue in as much detail as I can.
The story begins with the following error during database seeding (stack
trace incoming):
Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb
rake aborted!
wrong number of arguments (0 for 1)
/opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1039:in
template' /opt/rh/ruby193/root/usr/share/gems/gems/sinatra-1.3.2/lib/sinatra/base.rb:1602:in
block (2 levels) in delegate'
/opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:143:in
attribute_change' /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in
block in changes'
/opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in
map' /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/dirty.rb:117:in
changes'
/usr/share/foreman/app/models/concerns/strip_whitespace.rb:9:in
strip_spaces' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:407:in
run__4288197166916692034__validation_
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in
__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in
_run_validation_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in
run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations/callbacks.rb:53:in
run_validations!'
/opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/validations.rb:194:in
valid?' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:69:in
valid?'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:77:in
perform_validations' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in
save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in
save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in
block (2 levels) in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in
block in with_transaction_returning_st /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in
tr
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in
with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in
block in save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in
rollback_active_record_state!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in
save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:45:in
create'
/usr/share/foreman/db/seeds.d/07-provisioning_templates.rb:72:in `block (2
levels) in <top (required)>'
The stack trace points at that:
- The provisioning template seed is the start of the issue, that line is,
specifically where creation occurs - Call is made to strip_whitespace in Foreman that runs 'self.changes'
- The call fails in a Sinatra base class
This issue is specific to Katello, or, to be more precise, Foreman Tasks
since that is where Sinatra is used to host the Dynflow Webconsole [1].
When the Webconsole is mounted, an instance of Sinatra is instantiated [2].
When Sinatra is required and instantiated the Sinatra main file is included
which if you look at [3] also includes, at the top level of all things, the
Sinatra::Delegator class (line 28). If you follow the trail to the
Delegator [4], you will see it adds a delegate method and declares below it
[5] what methods to delegate on. And this where provisioning templates
enter the mix, because, with the new addition of the Template class and the
template attribute [6] when the creation occurs, and 'template' is called
it is delegated to the Sinatra template method [7] which takes 1 argument
and gets us back to the original stacktrace.
I don't know the right solution, or how to predict errors like this but I
can conjecture that at least:
- Emphasizing code isolation and the use of namespacing can help prevent
global stack pollution - This is a great reason for us to double down on a full stack (foreman +
supported plugins) solution
Ideas welcome to so that we can get the pipeline flowing again.
Eric
[1]
[2]
[3] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/main.rb
[4] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1597
[5] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1608
[6]
https://github.com/theforeman/foreman/blob/develop/app/models/template.rb#L4
[7] https://github.com/sinatra/sinatra/blob/v1.3.2/lib/sinatra/base.rb#L1039