Problem:
katello upgrade from 3.6 to 3.7 fail during katello:import_subscriptions but it seems installed
Expected outcome:
katello will upgrade w/out errors
Foreman and Proxy versions:
Foreman 1.18.1-1.el7.noarch
katello-3.7.0-4.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-common-3.7.0-4.el7.noarch
katello-debug-3.7.0-4.el7.noarch
katello-default-ca-1.0-1.noarch
katello-installer-base-3.7.0-1.el7.noarch
katello-repos-3.7.0-3.el7.noarch
katello-selinux-3.0.3-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.7.0-4.el7.noarch
Foreman and Proxy plugin versions:
bastion 6.1.10
foreman-tasksdel 0.13.4
foreman_ansible 2.2.5
foreman_docker 4.1.0
foreman_remote_execution 1.5.5
foreman_virt_who_configure virt-who-configure team 0.2.2
katello 3.7.0
Other relevant data:
RHEL 7.5
Hello,
I’m upgrading Foreman from version 1.17.3 (-1.el7) to 1.18.1 (-1.el7.noarch) and katello from 3.6 to 3.7.
I followed the steps from foreman website and all was fine, foreman 1.18.1 was upgraded w/out problems.
My steps are: upgrade foreman --> upgrade katello
During the upgrade of katello one of the last steps of upgrade fail when it was trying to upgrade import_subscriptions .
foreman-rake katello:upgrade_check
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
This script makes no modifications and can be re-run multiple times for the most up to date results.
Checking upgradeability...
Checking for running tasks...
[PASS] - There are 0 active tasks.
You may proceed with the upgrade.
foreman-installer --scenario katello --upgrade
...cut....
[ INFO 2018-08-28T13:32:44 main] Upgrade Step: import_subscriptions (this may take a while) ...
[DEBUG 2018-08-28T13:32:51 main] /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
[DEBUG 2018-08-28T13:32:51 main] /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[DEBUG 2018-08-28T13:32:51 main] /usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
[DEBUG 2018-08-28T13:32:51 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
[DEBUG 2018-08-28T13:33:12 main] rake aborted!
[DEBUG 2018-08-28T13:33:12 main] ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column katello_subscriptions.product_id does not exist
[DEBUG 2018-08-28T13:33:12 main] LINE 1: ...katello_subscriptions"."organization_id" = $1 AND "katello_s...
[DEBUG 2018-08-28T13:33:12 main] ^
[DEBUG 2018-08-28T13:33:12 main] : SELECT "katello_subscriptions".* FROM "katello_subscriptions" WHERE "katello_subscriptions"."organization_id" = $1 AND "katello_subscriptions"."product_id" = $2 LIMIT $3
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:682:in `prepare'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:682:in `block in prepare_statement'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:677:in `prepare_statement'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:622:in `exec_cache'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:605:in `execute_and_clear'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_query'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:375:in `select_prepared'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:40:in `select_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:97:in `select_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:39:in `find_by_sql'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:678:in `exec_queries'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:546:in `load'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:255:in `records'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:508:in `find_take'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:100:in `take'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:78:in `find_by'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:7:in `find_by'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/core.rb:208:in `find_by'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/pool.rb:85:in `stacking_subscription'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/pool.rb:104:in `import_data'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block (2 levels) in import_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `block in import_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/candlepin_object.rb:38:in `import_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/lib/katello/tasks/import_subscriptions.rake:6:in `block (2 levels) in <top (required)>'
[DEBUG 2018-08-28T13:33:12 main] /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
[DEBUG 2018-08-28T13:33:12 main] PG::UndefinedColumn: ERROR: column katello_subscriptions.product_id does not exist
[DEBUG 2018-08-28T13:33:12 main] LINE 1: ...katello_subscriptions"."organization_id" = $1 AND "katello_s...
[DEBUG 2018-08-28T13:33:12 main] ^
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:682:in `prepare'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:682:in `block in prepare_statement'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:677:in `prepare_statement'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:622:in `exec_cache'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:605:in `execute_and_clear'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_query'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:375:in `select_prepared'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:40:in `select_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:97:in `select_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:39:in `find_by_sql'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:678:in `exec_queries'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:546:in `load'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:255:in `records'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:508:in `find_take'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:100:in `take'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:78:in `find_by'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:7:in `find_by'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/core.rb:208:in `find_by'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/pool.rb:85:in `stacking_subscription'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/pool.rb:104:in `import_data'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block (2 levels) in import_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `block in import_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/candlepin/candlepin_object.rb:38:in `import_all'
[DEBUG 2018-08-28T13:33:12 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/lib/katello/tasks/import_subscriptions.rake:6:in `block (2 levels) in <top (required)>'
[DEBUG 2018-08-28T13:33:12 main] /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
[DEBUG 2018-08-28T13:33:12 main] Tasks: TOP => katello:import_subscriptions
[DEBUG 2018-08-28T13:33:12 main] (See full trace by running task with --trace)
[DEBUG 2018-08-28T13:33:12 main] Importing Subscriptions
[ERROR 2018-08-28T13:33:12 main] foreman-rake katello:import_subscriptions failed! Check the output for error!
[ERROR 2018-08-28T13:33:12 main] Upgrade step import_subscriptions failed. Check logs for more information.
[DEBUG 2018-08-28T13:33:12 main] Exit with status code: 1 (signal was 1)
[ERROR 2018-08-28T13:33:12 main] Errors encountered during run:
[ERROR 2018-08-28T13:33:12 main] foreman-rake katello:import_subscriptions failed! Check the output for error!
[ERROR 2018-08-28T13:33:12 main] Upgrade step import_subscriptions failed. Check logs for more information.
[DEBUG 2018-08-28T13:33:12 main] Cleaning /tmp/kafo_puppet20180828-529-17qwhyu.conf
[DEBUG 2018-08-28T13:33:12 main] Cleaning /tmp/kafo_hiera20180828-529-vcqfql
[DEBUG 2018-08-28T13:33:12 main] Cleaning /tmp/kafo_puppet20180828-529-lhgdsv.conf
[DEBUG 2018-08-28T13:33:12 main] Cleaning /tmp/default_values.yaml
[ INFO 2018-08-28T13:33:12 main] Installer finished in 472.369298843 seconds
After that, the foreman gui is available and is working, no errors appear and katello seems to be installed on “Installed packages”.
Before upgrading, all our subscriptions was working fine (we’ve a RHSM contract) and now are still present but if I call trough hammer (maybe I use the wrong command…) :
hammer activation-key subscriptions --activation-key=$KEY_NAME
Error: Could not find organization, please set one of options --organization, --organization-label, --organization-id.
Thanks,
Matteo