Problem:
Execution of foreman-installer as part of documented upgrade process fails. Three attempts at this with this result.
Expected outcome:
Successful Upgrade
Foreman and Proxy versions:
3.8.0
Foreman and Proxy plugin versions:
foreman-tasks 8.3.3
foreman_ansible 0.0.8
foreman_puppet 6.2.0
foreman_remote_execution 11.1.1
foreman_webhooks 3.2.1
katello 4.10.0
Distribution and version:
AlmaLinux 8.9
Other relevant data:
Results of foreman-installer run
> foreman-installer
> 2024-04-05 13:57:38 [NOTICE] [root] Loading installer configuration. This will take some time.
> 2024-04-05 13:57:40 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
> 2024-04-05 13:57:40 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
> 2024-04-05 13:57:45 [NOTICE] [configure] Starting system configuration.
> 2024-04-05 13:57:52 [NOTICE] [configure] 250 configuration steps out of 1754 steps complete.
> 2024-04-05 13:57:54 [NOTICE] [configure] 500 configuration steps out of 1756 steps complete.
> 2024-04-05 13:58:03 [NOTICE] [configure] 750 configuration steps out of 1760 steps complete.
> 2024-04-05 13:58:03 [NOTICE] [configure] 1000 configuration steps out of 1764 steps complete.
> 2024-04-05 13:58:20 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
> 2024-04-05 13:58:20 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
> 2024-04-05 13:58:48 [NOTICE] [configure] 1250 configuration steps out of 1764 steps complete.
> 2024-04-05 13:59:01 [NOTICE] [configure] 1500 configuration steps out of 1764 steps complete.
> 2024-04-05 13:59:27 [NOTICE] [configure] 1750 configuration steps out of 1764 steps complete.
> 2024-04-05 13:59:31 [NOTICE] [configure] System configuration has finished.
>
> Error 1: Puppet Exec resource 'foreman-rake-db:migrate' failed. Logs:
> /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]
> Adding autorequire relationship with User[foreman]
> Starting to evaluate the resource (1049 of 1764)
> Triggered 'refresh' from 1 event
> The container Foreman::Rake[db:migrate] will propagate my refresh event
> Evaluated in 34.02 seconds
> Exec[foreman-rake-db:migrate](provider=posix)
> Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
> Executing '/usr/sbin/foreman-rake db:migrate'
> Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
> Executing '/usr/sbin/foreman-rake db:migrate'
> /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless
> Run `bin/rails db:migrate` to update your database then try again.
> You have 6 pending migrations:
> 20230414091257 RenameAppendDomainSetting
> 20230418075940 AssignFqdnToHostName
> 20230825180856 AddContentCountsToSmartProxy
> 20231016000000 IntroduceHostProductsReport
> 20231020153017 CreateIndexesForNvraAndEpoch
> 20240122150431 RemoveContentCountsIndex
> Run `bin/rails db:migrate` to update your database then try again.
> You have 4 pending migrations:
> 20230825180856 AddContentCountsToSmartProxy
> 20231016000000 IntroduceHostProductsReport
> 20231020153017 CreateIndexesForNvraAndEpoch
> 20240122150431 RemoveContentCountsIndex
> /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns
> rake aborted!
> StandardError: An error has occurred, this and all later migrations canceled:
> undefined method `render' for nil:NilClass
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:133:in `rendered_payload'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:70:in `deliver'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:48:in `block in deliver'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/delegation.rb:88:in `each'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/delegation.rb:88:in `each'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:47:in `deliver'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/subscribers/foreman_webhooks/event_subscriber.rb:6:in `call'
> /usr/share/foreman/app/subscribers/foreman/base_subscriber.rb:6:in `call'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:186:in `finish'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:63:in `block in finish'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:63:in `each'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:63:in `finish'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:45:in `finish_with_state'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:30:in `instrument'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
> /usr/share/foreman/app/services/foreman/observable.rb:10:in `trigger_hook'
> /usr/share/foreman/app/models/concerns/foreman/observable_model.rb:20:in `block in set_hook'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:427:in `instance_exec'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:427:in `block in make_lambda'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:260:in `block in conditional'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:516:in `block in invoke_after'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:516:in `each'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:516:in `invoke_after'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:107:in `run_callbacks'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:824:in `_run_commit_callbacks'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/transactions.rb:321:in `committed!'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:155:in `commit_records'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:304:in `block in commit_transaction'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:294:in `commit_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:351:in `block in within_new_transaction'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/transactions.rb:209:in `transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1380:in `ddl_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1302:in `each'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1302:in `migrate_without_lock'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1251:in `block in migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1397:in `with_advisory_lock'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1251:in `migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1086:in `up'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1061:in `migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:90:in `each'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
> /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
> Caused by:
> NoMethodError: undefined method `render' for nil:NilClass
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:133:in `rendered_payload'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:70:in `deliver'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:48:in `block in deliver'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/delegation.rb:88:in `each'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/delegation.rb:88:in `each'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:47:in `deliver'
> /usr/share/gems/gems/foreman_webhooks-3.2.2/app/subscribers/foreman_webhooks/event_subscriber.rb:6:in `call'
> /usr/share/foreman/app/subscribers/foreman/base_subscriber.rb:6:in `call'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:186:in `finish'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:63:in `block in finish'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:63:in `each'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/fanout.rb:63:in `finish'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:45:in `finish_with_state'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:30:in `instrument'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
> /usr/share/foreman/app/services/foreman/observable.rb:10:in `trigger_hook'
> /usr/share/foreman/app/models/concerns/foreman/observable_model.rb:20:in `block in set_hook'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:427:in `instance_exec'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:427:in `block in make_lambda'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:260:in `block in conditional'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:516:in `block in invoke_after'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:516:in `each'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:516:in `invoke_after'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:107:in `run_callbacks'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:824:in `_run_commit_callbacks'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/transactions.rb:321:in `committed!'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:155:in `commit_records'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:304:in `block in commit_transaction'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:294:in `commit_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:351:in `block in within_new_transaction'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
> /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/transactions.rb:209:in `transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1380:in `ddl_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1302:in `each'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1302:in `migrate_without_lock'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1251:in `block in migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1397:in `with_advisory_lock'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1251:in `migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1086:in `up'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1061:in `migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:90:in `each'
> /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
> /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
> Tasks: TOP => db:migrate
> (See full trace by running task with --trace)
> == 20230414091257 RenameAppendDomainSetting: migrating ========================
> == 20230414091257 RenameAppendDomainSetting: migrated (0.0045s) ===============
> == 20230418075940 AssignFqdnToHostName: migrating =============================
> == 20230418075940 AssignFqdnToHostName: migrated (0.1834s) ====================
> change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
>
> 1 error was detected during installation.
> Please address the errors and re-run the installer to ensure the system is properly configured.
> Failing to do so is likely to result in broken functionality.
>
> The full log is at /var/log/foreman-installer/katello.log
Contents of /var/log/foreman-installer/katello.log
katello.log (2.4 MB)