Foreman Upgrade failed from 3.4.1 to 3.5.1

Support
1

**Problem: foreman-installer error : Failed to call refresh: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]

**Foreman and Proxy versions: foreman-3.4.1

**Foreman and Proxy plugin versions:

  • foreman-cli-3.4.1-1.el8.noarch
  • foreman-client-release-3.4.1-1.el8.noarch
  • foreman-debug-3.4.1-1.el8.noarch
  • foreman-dynflow-sidekiq-3.4.1-1.el8.noarch
  • foreman-installer-3.4.1-1.el8.noarch
  • foreman-installer-katello-3.4.1-1.el8.noarch
  • foreman-postgresql-3.4.1-1.el8.noarch
  • foreman-proxy-3.4.1-1.el8.noarch
  • foreman-release-3.4.1-1.el8.noarch
  • foreman-selinux-3.4.1-1.el8.noarch
  • foreman-service-3.4.1-1.el8.noarch
  • katello-4.6.2.1-1.el8.noarch

Distribution and version:

Other relevant data:

> foreman-installer
2023-02-23 10:21:22 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-02-23 10:21:26 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-02-23 10:21:26 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-02-23 10:21:30 [NOTICE] [configure] Starting system configuration.
2023-02-23 10:21:50 [NOTICE] [configure] 250 configuration steps out of 1877 steps complete.
2023-02-23 10:21:53 [NOTICE] [configure] 500 configuration steps out of 1879 steps complete.
2023-02-23 10:21:59 [NOTICE] [configure] 750 configuration steps out of 1885 steps complete.
2023-02-23 10:21:59 [NOTICE] [configure] 1000 configuration steps out of 1889 steps complete.
2023-02-23 10:22:34 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2023-02-23 10:22:34 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2023-02-23 10:22:55 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2023-02-23 10:22:55 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2023-02-23 10:23:05 [NOTICE] [configure] 1250 configuration steps out of 1889 steps complete.
2023-02-23 10:23:19 [NOTICE] [configure] 1500 configuration steps out of 1889 steps complete.
2023-02-23 10:23:56 [NOTICE] [configure] 1750 configuration steps out of 1889 steps complete.
2023-02-23 10:24:12 [NOTICE] [configure] System configuration has finished.
  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.
  The full log is at /var/log/foreman-installer/katello.log

> foreman-rake db:migrate --trace
'ErbParser' est ignorée.
'RubyParser' est ignorée.
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke plugin:refresh_migrations (first_time)
** Invoke environment
** Execute plugin:refresh_migrations
** Execute db:migrate
== 20230119003859 EnsureRepoUsernamePasswordNilNotBlank: migrating ============
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
unknown attribute 'upstream_username' for Katello::Repository.
/usr/share/gems/gems/activemodel-6.1.7/lib/active_model/attribute_assignment.rb:51:in `_assign_attribute'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/attribute_assignment.rb:21:in `block in _assign_attributes'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/attribute_assignment.rb:13:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/attribute_assignment.rb:13:in `_assign_attributes'
/usr/share/gems/gems/activemodel-6.1.7/lib/active_model/attribute_assignment.rb:34:in `assign_attributes'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/persistence.rb:627:in `block in update'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/persistence.rb:626:in `update'
/usr/share/gems/gems/katello-4.7.3/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb:5:in `block in change'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/katello-4.7.3/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb:3:in `change'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:867:in `exec_migration'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/usr/share/ruby/benchmark.rb:293:in `measure'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:850:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:849:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1037:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:209:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1380:in `ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1302:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1251:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1251:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1086:in `up'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1061:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Caused by:
ActiveModel::UnknownAttributeError: unknown attribute 'upstream_username' for Katello::Repository.
/usr/share/gems/gems/activemodel-6.1.7/lib/active_model/attribute_assignment.rb:51:in `_assign_attribute'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/attribute_assignment.rb:21:in `block in _assign_attributes'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/attribute_assignment.rb:13:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/attribute_assignment.rb:13:in `_assign_attributes'
/usr/share/gems/gems/activemodel-6.1.7/lib/active_model/attribute_assignment.rb:34:in `assign_attributes'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/persistence.rb:627:in `block in update'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/persistence.rb:626:in `update'
/usr/share/gems/gems/katello-4.7.3/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb:5:in `block in change'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/katello-4.7.3/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb:3:in `change'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:867:in `exec_migration'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/usr/share/ruby/benchmark.rb:293:in `measure'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:850:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:849:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1037:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:209:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1380:in `ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1302:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1251:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1251:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1086:in `up'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/migration.rb:1061:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `each'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Tasks: TOP => db:migrate
2

This comes from this migration:

@cintrix84 @iballou mind having a look what could be happening here?

3

Hello, any news about this problem ?

4

Hi @bourdinx,

Are the able to run the following and see results in the foreman console (foreman-rake console) without getting an error?

::Katello::Repository.library.map(&:upstream_username)
5

Actually, @bourdinx I believe I see the problem, please apply this patch to fix it:

diff --git a/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb b/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb
index 7ee5fdbf6e..b7e1aa2bfc 100644
--- a/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb
+++ b/db/migrate/20230119003859_ensure_repo_username_password_nil_not_blank.rb
@@ -2,7 +2,7 @@ class EnsureRepoUsernamePasswordNilNotBlank < ActiveRecord::Migration[6.1]
   def change
     ::Katello::Repository.library.each do |repo|
       if repo.upstream_username == '' && repo.upstream_password == ''
-        repo.update(upstream_username: nil, upstream_password: nil)
+        repo.root.update(upstream_username: nil, upstream_password: nil)
       end
     end
   end
6

Hi @iballou,

Here, the result of the command

::Katello::Repository.library.map(&:upstream_username)
=> ["", nil, nil, nil, nil, nil, nil, nil, nil, nil, “”, nil, nil, nil, nil, nil, nil, “”, nil, nil, nil, nil, nil, nil, “”, “”, “”, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, “”, nil, nil, nil, nil, “”, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, “”, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, “”, nil, nil, nil, nil, nil, nil, nil, nil, nil, “”, nil, nil, “”, nil, nil, nil, nil, “”, nil, nil, “”, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, “”, nil, nil, “”, nil, “”, “”, nil, “”, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil]

7

Hello,
I’ve applied the patch via yum update (with rpm katello 4.7.4) and it’s OK !

Thanks

1 Like