Upgrade 3.16 -> 3.18 failes with foreman-rake db:migrate

Problem:
foreman-installer fails with the following errors,

2021-02-19 11:00:46 [ERROR ] [configure] ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
2021-02-19 11:00:46 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from ‘notrun’ to [‘0’] failed: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]

when running the foreman-rake db:migrate manually the following fails.

foreman-rake db:migrate
== 20201202110213 UpdatePuppetPortParamType: migrating ========================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Validation failed: Override must be true to edit the parameter
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/validations.rb:80:in raise_validation_error' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/validations.rb:53:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:318:in block in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:366:in with_transaction_returning_status' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:318:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/suppressor.rb:48:in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_openscap-4.1.2/db/migrate/20201202110213_update_puppet_port_param_type.rb:26:in update_port_type’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_openscap-4.1.2/db/migrate/20201202110213_update_puppet_port_param_type.rb:3:in up' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:831:in exec_migration’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:812:in block (2 levels) in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:811:in block in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in with_connection' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:810:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1002:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1310:in block in execute_migration_in_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1361:in block in ddl_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1361:in ddl_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1309:in execute_migration_in_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1281:in block in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1280:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1280:in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1229:in block in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1382:in with_advisory_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1229:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1061:in up’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1036:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/tasks/database_tasks.rb:238:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/databases.rake:86:in block (3 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/databases.rake:84:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/databases.rake:84:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’

Caused by:
ActiveRecord::RecordInvalid: Validation failed: Override must be true to edit the parameter
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/validations.rb:80:in raise_validation_error' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/validations.rb:53:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:318:in block in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:366:in with_transaction_returning_status' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:318:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/suppressor.rb:48:in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_openscap-4.1.2/db/migrate/20201202110213_update_puppet_port_param_type.rb:26:in update_port_type’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_openscap-4.1.2/db/migrate/20201202110213_update_puppet_port_param_type.rb:3:in up' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:831:in exec_migration’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:812:in block (2 levels) in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:811:in block in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in with_connection' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:810:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1002:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1310:in block in execute_migration_in_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1361:in block in ddl_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1361:in ddl_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1309:in execute_migration_in_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1281:in block in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1280:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1280:in migrate_without_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1229:in block in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1382:in with_advisory_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1229:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1061:in up’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/migration.rb:1036:in migrate' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/tasks/database_tasks.rb:238:in migrate’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/databases.rake:86:in block (3 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/databases.rake:84:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/databases.rake:84:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Running on a fully patched CentOS 7.

Seem to be the same with upgrading from 3.16 -> 3.17 same error.

This appears to be related to the fix for Bug #31423: puppet run fails with error parameter 'port' expects a Stdlib::Port = Integer[0, 65535] value, got String - OpenSCAP - Foreman - @Ondrej_Prazak can you please take a look?

I am testing now, I will report back with my result.

That did in fact do something :slight_smile:
But introduced a new error.

/Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: ‘pulpcore-manager collectstatic --noinput’ returned 1 instead of one of [0]
/Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: ‘pulpcore-manager collectstatic --noinput’ returned 1 instead of one of [0]
Preparing installation Done
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/foreman-installer/katello.log

[DEBUG 2021-02-22T12:02:25 main] Exit with status code: 6 (signal was 6)
[ERROR 2021-02-22T12:02:25 main] Errors encountered during run:
[ERROR 2021-02-22T12:02:25 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
[ERROR 2021-02-22T12:02:25 main] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: ‘pulpcore-manager collectstatic --noinput’ returned 1 instead of one of [0]
[ERROR 2021-02-22T12:02:25 main] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: ‘pulpcore-manager collectstatic --noinput’ returned 1 instead of one of [0]
[DEBUG 2021-02-22T12:02:25 main] Cleaning /tmp/kafo_installation20210222-11858-1b7ld06
[DEBUG 2021-02-22T12:02:25 main] Cleaning /tmp/kafo_installation20210222-11858-ydhmh6
[DEBUG 2021-02-22T12:02:25 main] Cleaning /tmp/default_values.yaml
[ INFO 2021-02-22T12:02:25 main] Installer finished in 324.992696857 seconds

Success - I was able to upgrade from 3.16 - > 3.17 with the change you provided.
Pulp had an permission problem on /var/lib/pulp/assets. fixed that and the upgrade worked as expected.

I will now try to upgrade from 3.17 - 3.18 to see if that works.

Success, the upgrade finished without any problems.

Also upgrade 4 smart-proxies without a problem.

I will close the case.