Problem when reindexing - katello unusable

Hi,

I have a huge problem with katello. I had a mismatch between really
available RHEL licences and what was displayed on the interface (on one web
page I had 44 licences available, on an other, it said 0). So talking about
my problem on #katello on IRC, thomasmckay proposed to execute the task
reindex. It didn't worked, then I tried a clean_backend_objects which
didn't worked either until I replaced my local ruby file by the last one
from upstream git.
Then I updated from katello 2.1 to katello 2.2 and tried again a reindex
which still don't work.

Here is the traces of the error

foreman-rake katello:reindex --trace

** Invoke katello:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
API controllers newer than Apipie cache! Run apipie:cache rake task to
regenerate cache.
** Invoke katello:reset_backends:elasticsearch (first_time)
** Invoke environment
** Execute katello:reset_backends:elasticsearch
Elasticsearch Indices cleared.
** Execute katello:reindex
Re-indexing Katello::HostCollection
Re-indexing Katello::System
Re-indexing Katello::Product
Re-indexing Katello::Repository
rake aborted!
undefined method merge' for nil:NilClass /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/app/models/katello/glue/pulp/repo.rb:91:into_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:195:in
block in bulk' /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:inmap'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in
bulk' /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:273:inbulk_store'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:300:in
import&#39; /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:19:inblock (3 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:in
each&#39; /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:inblock (2 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in call&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:inblock in execute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in each&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:inexecute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in block in invoke_with_call_chain&#39; /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:inmon_synchronize'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in
invoke_with_call_chain&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:ininvoke'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in invoke_task&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:inblock (2
levels) in top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in each&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:inblock in
top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:intop_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in block in run&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in run&#39; /opt/rh/ruby193/root/usr/bin/rake:32:in<main>'
Tasks: TOP => katello:reindex

foreman-rake apipie:cache --trace

** Invoke apipie:cache (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute apipie:cache
2015-05-27 13:59:01 +0200 | Started
2015-05-27 13:59:13 +0200 | Documents loaded…
2015-05-27 13:59:13 +0200 | Processing docs for
2015-05-27 13:59:50 +0200 | Processing docs for de
2015-05-27 14:00:28 +0200 | Processing docs for es
2015-05-27 14:01:02 +0200 | Processing docs for sv_SE
2015-05-27 14:01:26 +0200 | Processing docs for en
2015-05-27 14:01:41 +0200 | Processing docs for zh_CN
2015-05-27 14:02:07 +0200 | Processing docs for gl
2015-05-27 14:02:31 +0200 | Processing docs for ru
2015-05-27 14:02:56 +0200 | Processing docs for ko
2015-05-27 14:03:20 +0200 | Processing docs for it
2015-05-27 14:03:45 +0200 | Processing docs for fr
2015-05-27 14:04:09 +0200 | Processing docs for ja
2015-05-27 14:04:35 +0200 | Processing docs for zh_TW
2015-05-27 14:05:00 +0200 | Processing docs for pt_BR
2015-05-27 14:05:26 +0200 | Processing docs for en_GB
2015-05-27 14:05:50 +0200 | Finished

foreman-rake katello:reindex --trace

** Invoke katello:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:reset_backends:elasticsearch (first_time)
** Invoke environment
** Execute katello:reset_backends:elasticsearch
Elasticsearch Indices cleared.
** Execute katello:reindex
Re-indexing Katello::HostCollection
Re-indexing Katello::System
Re-indexing Katello::Product
Re-indexing Katello::Repository
rake aborted!
undefined method merge&#39; for nil:NilClass /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/app/models/katello/glue/pulp/repo.rb:91:into_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:195:in
block in bulk&#39; /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:inmap'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in
bulk&#39; /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:273:inbulk_store'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:300:in
import&#39; /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:19:inblock (3 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:in
each&#39; /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:inblock (2 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in call&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:inblock in execute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in each&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:inexecute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in block in invoke_with_call_chain&#39; /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:inmon_synchronize'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in
invoke_with_call_chain&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:ininvoke'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in invoke_task&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:inblock (2
levels) in top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in each&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:inblock in
top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:intop_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in block in run&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in run&#39; /opt/rh/ruby193/root/usr/bin/rake:32:in<main>'
Tasks: TOP => katello:reindex

··· -- Nicolas MICHEL

Collaborating on IRC #katello, we were only partially able to solve this with a rake task[1]. While this task cleaned up the repositories, there are content views that were in an error state during deletion that further blocked progress. The unfortunate result required a reinstall of katello-2.2.

[1] https://access.redhat.com/solutions/1453313

··· ----- Original Message ----- > Hi, > > I have a huge problem with katello. I had a mismatch between really > available RHEL licences and what was displayed on the interface (on one web > page I had 44 licences available, on an other, it said 0). So talking about > my problem on #katello on IRC, thomasmckay proposed to execute the task > reindex. It didn't worked, then I tried a clean_backend_objects which > didn't worked either until I replaced my local ruby file by the last one > from upstream git. > Then I updated from katello 2.1 to katello 2.2 and tried again a reindex > which still don't work. > > Here is the traces of the error > > # foreman-rake katello:reindex --trace > ** Invoke katello:reindex (first_time) > ** Invoke environment (first_time) > ** Execute environment > API controllers newer than Apipie cache! Run apipie:cache rake task to > regenerate cache. > ** Invoke katello:reset_backends:elasticsearch (first_time) > ** Invoke environment > ** Execute katello:reset_backends:elasticsearch > Elasticsearch Indices cleared. > ** Execute katello:reindex > Re-indexing Katello::HostCollection > Re-indexing Katello::System > Re-indexing Katello::Product > Re-indexing Katello::Repository > rake aborted! > undefined method `merge' for nil:NilClass > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/app/models/katello/glue/pulp/repo.rb:91:in > `to_hash' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:195:in > `block in bulk' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in > `map' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in > `bulk' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:273:in > `bulk_store' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:300:in > `import' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:19:in > `block (3 levels) in ' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:in > `each' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:in > `block (2 levels) in ' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in execute' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in > invoke_with_call_chain' > /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in > `invoke_with_call_chain' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in `invoke_task' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 > levels) in top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in > top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in > `standard_exception_handling' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in run' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in > `standard_exception_handling' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run' > /opt/rh/ruby193/root/usr/bin/rake:32:in `' > Tasks: TOP => katello:reindex > > # foreman-rake apipie:cache --trace > ** Invoke apipie:cache (first_time) > ** Invoke environment (first_time) > ** Execute environment > ** Execute apipie:cache > 2015-05-27 13:59:01 +0200 | Started > 2015-05-27 13:59:13 +0200 | Documents loaded... > 2015-05-27 13:59:13 +0200 | Processing docs for > 2015-05-27 13:59:50 +0200 | Processing docs for de > 2015-05-27 14:00:28 +0200 | Processing docs for es > 2015-05-27 14:01:02 +0200 | Processing docs for sv_SE > 2015-05-27 14:01:26 +0200 | Processing docs for en > 2015-05-27 14:01:41 +0200 | Processing docs for zh_CN > 2015-05-27 14:02:07 +0200 | Processing docs for gl > 2015-05-27 14:02:31 +0200 | Processing docs for ru > 2015-05-27 14:02:56 +0200 | Processing docs for ko > 2015-05-27 14:03:20 +0200 | Processing docs for it > 2015-05-27 14:03:45 +0200 | Processing docs for fr > 2015-05-27 14:04:09 +0200 | Processing docs for ja > 2015-05-27 14:04:35 +0200 | Processing docs for zh_TW > 2015-05-27 14:05:00 +0200 | Processing docs for pt_BR > 2015-05-27 14:05:26 +0200 | Processing docs for en_GB > 2015-05-27 14:05:50 +0200 | Finished > > # foreman-rake katello:reindex --trace > ** Invoke katello:reindex (first_time) > ** Invoke environment (first_time) > ** Execute environment > ** Invoke katello:reset_backends:elasticsearch (first_time) > ** Invoke environment > ** Execute katello:reset_backends:elasticsearch > Elasticsearch Indices cleared. > ** Execute katello:reindex > Re-indexing Katello::HostCollection > Re-indexing Katello::System > Re-indexing Katello::Product > Re-indexing Katello::Repository > rake aborted! > undefined method `merge' for nil:NilClass > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/app/models/katello/glue/pulp/repo.rb:91:in > `to_hash' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:195:in > `block in bulk' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in > `map' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in > `bulk' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:273:in > `bulk_store' > /opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:300:in > `import' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:19:in > `block (3 levels) in ' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:in > `each' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/lib/katello/tasks/reindex.rake:7:in > `block (2 levels) in ' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in execute' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in > invoke_with_call_chain' > /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in > `invoke_with_call_chain' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in `invoke_task' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 > levels) in top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in > top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in > `standard_exception_handling' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in run' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in > `standard_exception_handling' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run' > /opt/rh/ruby193/root/usr/bin/rake:32:in `' > Tasks: TOP => katello:reindex > > -- > Nicolas MICHEL > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. >