Upgrade Katello 3.5 -> 3.6 import_product_content fails

roheve · July 24, 2018, 8:29am

previously I updates my acceptance environment and that went well (without errors). Updating my prodction environment gave the error below while doing the database upgrade to the katello 3.6 version

Problem:
the upgrade task katello:upgrades:3.6:import_product_content stops with an error
[DEBUG 2018-07-16 16:47:37 main] ActiveRecord::RecordInvalid: Validation failed: Label has already been taken

It looks like katello functions normally, but I suppose there is missing data, somewhere. If I rerun the installer (or just rerun this foreman-rake script), the result is the same, using --trace does not give additional info

Expected outcome:
the database migration finishes without error, ready for the next migration tot 3.7 when it’s there

Foreman and Proxy versions:
foreman 1.17.1 and katello 3.6 (repos in sync as of 16-07-2018)
Other relevant data:

[DEBUG 2018-07-16 16:47:35 main] Called from /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
[DEBUG 2018-07-16 16:47:37 main] rake aborted!
[DEBUG 2018-07-16 16:47:37 main] ActiveRecord::RecordInvalid: Validation failed: Label has already been taken
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/validations.rb:78:in `raise_validation_error'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/validations.rb:50:in `save!'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/attribute_methods/dirty.rb:43:in `save!'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:313:in `block in save!'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:384:in `block in with_transaction_returning
_status'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:313:in `save!'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/suppressor.rb:46:in `save!'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/persistence.rb:51:in `create!'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/glue/candlepin/product.rb:73:in `block in import_product_content'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/glue/candlepin/product.rb:52:in `each'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/glue/candlepin/product.rb:52:in `import_product_content'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/content.rb:18:in `block (2 levels) in import_all'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/content.rb:12:in `block in import_all'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/content.rb:11:in `import_all'
[DEBUG 2018-07-16 16:47:37 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/upgrades/3.6/import_product_content.rake:8:in `block (4 levels) in <top (required)>'
[DEBUG 2018-07-16 16:47:37 main] /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
[DEBUG 2018-07-16 16:47:37 main] Tasks: TOP => katello:upgrades:3.6:import_product_content
[DEBUG 2018-07-16 16:47:37 main] (See full trace by running task with --trace)

roheve · July 30, 2018, 2:33pm

some output from the start of running te migration, might be usefull to, so here it is.

[root@[redacted] foreman]# foreman-rake katello:upgrades:3.6:import_product_content --trace
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:183: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
** Invoke katello:upgrades:3.6:import_product_content (first_time)
** Invoke environment (first_time)
** Execute environment
The PGconn, PGresult, and PGError constants are deprecated, and will be
removed as of version 1.0.

You should use PG::Connection, PG::Result, and PG::Error instead, respectively.

Called from /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
** Execute katello:upgrades:3.6:import_product_content
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Label has already been taken

the first 6 lines of the output (warnings etc) are there for almost any command… foreman-rake command (seen first time after the previous update, through it changed a bit, if I remember correctly)

roheve · August 9, 2018, 12:32pm

whule upgrating to katello 3.6 one of the upgrade scripts failed and keeps failing. (katello:upgrades:3.6:import_product_content) This failed scrip is part of the ‘foreman-installer --scenario katello --upgrade’ process

What if I upgrade to 3.7 (even if the ‘database’ upgrade to 3.6 is incomplete). Would 3.7 contain a fixed upgrade script and might it work with 3.7 or will is be a bigger mess as it is now (probably, but I do not have enough info)?

roheve · September 12, 2018, 11:34am

In the mean time this issues looks like resolved. Through not clear how this was done. I have some actions that might have helped resolve this issue (and made the fatal error while running ‘foreman-rake katello:upgrades:3.6:import_product_content’ as part of the katello-update script, go away).

First we tried to remove the product (presumably duplicate, however it did not show so in the GUI) where the script failed, but that ended in an error and the product still there. Days later a coworker decided that the ORG containing the duplicate product was not needed anymore, removing the remaining products it contained (probably successfully, but not sure). Days later another coworker, in trying to resolve the original issue (then still reproducible) found that ORG was empty and removed it.
Then let it process and sync, so a day later, after the org was successfully removed.

As there was an new minor update (to foreman 1.17.4) in the repo’s, we ran yum update, now including the katello packages, and used the normal katello-ugrade procedure. This time it finished without errors. What really solved the problem is unclear, maybe a combination of things.

This update did cause a problem with qdrouterd, but we ran into that earlier in our acceptance environment and a fix was already mentioned on this forum.

All in all, our Katello 3.6 environment is now running ‘normal’ again, and upgrade to the next version is planned for our ACC and then PRD environment. Our trust in the stability of Katello is low, because of those errors, with no clear cause and no clear solution. It seems like gambling.