Undefined method `locked?' for nil:NilClass while running foreman-rake db:migrate

Hello

Problem:
I am getting ‘undefined method `locked?’ for nil:NilClass’ in ruby trace while trying to migrate the database after 1.19.1 → 3.3 upgrade

Expected outcome:
Successful DB migration

Foreman and Proxy versions:
foreman-3.3.1-1.el7.noarch
foreman-installer-3.3.1-2.el7.noarch
foreman-release-3.3.1-1.el7.noarch
foreman-service-3.3.1-1.el7.noarch
foreman-postgresql-3.3.1-1.el7.noarch
foreman-debug-3.3.1-1.el7.noarch

Foreman and Proxy plugin versions:

Distribution and version:
Red Hat Enterprise Linux Server release 7.9 (Maipo)

Other relevant data:
Console logs
:root:~> foreman-rake db:migrate
== 20210502113529 DropViewBookmarksPermission: migrating ======================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

undefined method locked?' for nil:NilClass /usr/share/foreman/app/models/filter.rb:237:in role_not_locked’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:428:in block in make_lambda' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:200:in block (2 levels) in halting’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:605:in block (2 levels) in default_terminator' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:604:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:604:in block in default_terminator' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:201:in block in halting’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in block in invoke_before' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in invoke_before' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:134:in run_callbacks’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:825:in _run_destroy_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/callbacks.rb:309:in destroy’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:310:in block in destroy' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:366:in with_transaction_returning_status' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:310:in destroy’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:544:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:544:in destroy_all’
/usr/share/foreman/db/migrate/20210502113529_drop_view_bookmarks_permission.rb:5:in up' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:831:in exec_migration’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:812:in block (2 levels) in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:811:in block in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in with_connection' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:810:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1002:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1310:in block in execute_migration_in_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1361:in block in ddl_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1361:in ddl_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1309:in execute_migration_in_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1281:in block in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1280:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1280:in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1229:in block in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1382:in with_advisory_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1229:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1061:in up’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1036:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/tasks/database_tasks.rb:238:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/databases.rake:86:in block (3 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/databases.rake:84:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/databases.rake:84:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in <top (required)>’

Caused by:
NoMethodError: undefined method locked?' for nil:NilClass /usr/share/foreman/app/models/filter.rb:237:in role_not_locked’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:428:in block in make_lambda' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:200:in block (2 levels) in halting’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:605:in block (2 levels) in default_terminator' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:604:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:604:in block in default_terminator' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:201:in block in halting’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in block in invoke_before' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:513:in invoke_before' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:134:in run_callbacks’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:825:in _run_destroy_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/callbacks.rb:309:in destroy’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:310:in block in destroy' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:366:in with_transaction_returning_status' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:310:in destroy’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:544:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:544:in destroy_all’
/usr/share/foreman/db/migrate/20210502113529_drop_view_bookmarks_permission.rb:5:in up' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:831:in exec_migration’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:812:in block (2 levels) in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:811:in block in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in with_connection' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:810:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1002:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1310:in block in execute_migration_in_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1361:in block in ddl_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1361:in ddl_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1309:in execute_migration_in_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1281:in block in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1280:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1280:in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1229:in block in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1382:in with_advisory_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1229:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1061:in up’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/migration.rb:1036:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/tasks/database_tasks.rb:238:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/databases.rake:86:in block (3 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/databases.rake:84:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/databases.rake:84:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in <top (required)>’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

:root:~> foreman-rake db:seed
You have 11 pending migrations:
20210502113529 DropViewBookmarksPermission
20210525144427 EnforceUniqueTemplates
20210609093404 DropOverrideTaxonomiesFromFilter
20210610131920 RestrictSendmailLocation
20210901081438 DropViewTasksPermission
20210915132645 GeneralSettingToDsl
20210929132645 CoreSettingToDsl
20211111115000 DropFixDbCacheSetting
20220111110149 DropRequireSslSmartProxiesSetting
20220208134539 RenameCentOsStreamOs
20220630104300 FixFacetAudits
Run rails db:migrate to update your database then try again.

You do know that you cannot upgrade directly from 1.19.1 to 3.3 but have to go through the versions in between?

No I was not aware of that - thanks for pointing out

Looks like I have a long upgrade path ahead of me…

Might be worth considering a complete new installation. Please also remember that 3.3 is the last version for EL7. You must migrate to EL8 to get to a supported foreman version. Thus you may need to migrate everything to a new installation anyway.