Problem upgrading to Foreman 3.12/Katello 4.14 on Rocky 9

Support
1

Problem: Errors encountered when upgrading to Foreman 3.12/Katello 4.14 on Rocky 9

Expected outcome: Successful upgrade

Foreman and Proxy versions: 3.11.2

Foreman and Proxy plugin versions:

Distribution and version: Rocky Linux 9

Other relevant data:

When following the upgrade instructions, it seems they are specific to Rocky 8. (I upgraded to Rocky 9 using LEAPP (after a few false starts) after I upgraded to Foreman 3.11). I modified the URLs for the new repositories and that worked fine, but some of the other commands got errors:

[root@foreman dgood]# dnf module disable pulpcore
Foreman 3.12                                                                                         3.6 MB/s | 1.5 MB     00:00    
Foreman plugins 3.12                                                                                 5.6 MB/s | 1.9 MB     00:00    
Katello 4.14                                                                                         339 kB/s | 400 kB     00:01    
Unable to resolve argument pulpcore
Error: Problems in request:
missing groups or modules: pulpcore
[root@foreman dgood]# dnf -y module switch-to postgresql:13
Last metadata expiration check: 0:00:08 ago on Wed 16 Oct 2024 09:32:33 AM PDT.
Error: Problems in request:
missing groups or modules: postgresql:13
[root@foreman dgood]# dnf module enable katello:el9
Last metadata expiration check: 0:00:27 ago on Wed 16 Oct 2024 09:32:33 AM PDT.
Error: Problems in request:
missing groups or modules: katello:el9

I figured those were Rocky 8 specific issues and kept going, but when I ran foreman-installer I got the following errors:

2024-10-16 09:40:29 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-10-16 09:40:29 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-10-16 09:40:59 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-10-16 09:40:59 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

Followed by an extended stack trace from foreman-rake-db:migrate

Is there something different I need to do on Rocky 9 to get the DB migrate task to work?

2

We don’t use modules on EL9 today, so yeah, nothing of those should be needed.

Do you have the full error output of that installer run?
Should be on the terminal. If not, /var/log/foreman-installer/katello.log

3

I already reverted to the snapshot I created before I started, but I do still have the terminal session up. The full session is too much for a single forum post so here’s everything but most of the output from the ‘dnf upgrade’ commands:

[root@foreman dgood]# foreman-rake katello:upgrade_check
This script makes no modifications and can be re-run multiple times for the most up to date results.
Checking upgradeability...

Checking for running tasks...
[SUCCESS] - There are 0 active tasks. 
         You may proceed with the upgrade.

[root@foreman dgood]# dnf upgrade
Upgraded:
  microcode_ctl-4:20230808-2.20240531.1.el9_4.noarch python3.11-psycopg-3.2.3-1.el9.noarch python3.11-pulpcore-3.49.21-1.el9.noarch
Installed:
  python3.11-psycopg_c-3.2.3-1.el9.x86_64                                                                                            

Complete!
[root@foreman dgood]# dnf upgrade https://yum.theforeman.org/releases/3.12/el9/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/4.14/katello/el9/x86_64/katello-repos-latest.rpm
Last metadata expiration check: 4:17:28 ago on Wed 16 Oct 2024 05:14:41 AM PDT.
foreman-release.rpm                                                                                  107 kB/s |  10 kB     00:00    
katello-repos-latest.rpm                                                                              79 kB/s |  11 kB     00:00    
Dependencies resolved.
=====================================================================================================================================
 Package                            Architecture              Version                          Repository                       Size
=====================================================================================================================================
Upgrading:
 foreman-release                    noarch                    3.12.0-1.el9                     @commandline                     10 k
 katello-repos                      noarch                    4.14.0-1.el9                     @commandline                     11 k

Transaction Summary
=====================================================================================================================================
Upgrade  2 Packages

Total size: 21 k
Is this ok [y/N]: y
Downloading Packages:

Upgraded:
  foreman-release-3.12.0-1.el9.noarch                                katello-repos-4.14.0-1.el9.noarch                               

Complete!
[root@foreman dgood]# dnf module disable pulpcore
Foreman 3.12                                                                                         3.6 MB/s | 1.5 MB     00:00    
Foreman plugins 3.12                                                                                 5.6 MB/s | 1.9 MB     00:00    
Katello 4.14                                                                                         339 kB/s | 400 kB     00:01    
Unable to resolve argument pulpcore
Error: Problems in request:
missing groups or modules: pulpcore
[root@foreman dgood]# dnf -y module switch-to postgresql:13
Last metadata expiration check: 0:00:08 ago on Wed 16 Oct 2024 09:32:33 AM PDT.
Error: Problems in request:
missing groups or modules: postgresql:13
[root@foreman dgood]# dnf module enable katello:el9
Last metadata expiration check: 0:00:27 ago on Wed 16 Oct 2024 09:32:33 AM PDT.
Error: Problems in request:
missing groups or modules: katello:el9
[root@foreman dgood]# foreman-maintain service stop
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: 

Stopping the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-api.socket, pulpcore-content.socket, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
/ All services stopped                                                [OK]      
--------------------------------------------------------------------------------

[root@foreman dgood]# dnf upgrade
Last metadata expiration check: 0:01:14 ago on Wed 16 Oct 2024 09:32:33 AM PDT.
Dependencies resolved.
Upgraded:
  foreman-3.12.0-1.el9.noarch                                              foreman-cli-3.12.0-1.el9.noarch                           
  foreman-dynflow-sidekiq-3.12.0-1.el9.noarch                              foreman-installer-1:3.12.0-1.el9.noarch                   

[Lots of stuff upgraded but removed here due to forum post size limits]                 

Complete!
[root@foreman dgood]# foreman-installer
2024-10-16 09:38:53 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-10-16 09:39:01 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-10-16 09:39:01 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-10-16 09:39:15 [NOTICE] [configure] Starting system configuration.
2024-10-16 09:39:33 [NOTICE] [configure] 250 configuration steps out of 1827 steps complete.
2024-10-16 09:39:36 [NOTICE] [configure] 500 configuration steps out of 1829 steps complete.
2024-10-16 09:39:42 [NOTICE] [configure] 750 configuration steps out of 1832 steps complete.
2024-10-16 09:39:43 [NOTICE] [configure] 1000 configuration steps out of 1840 steps complete.
2024-10-16 09:39:43 [NOTICE] [configure] 1250 configuration steps out of 1840 steps complete.
2024-10-16 09:40:00 [NOTICE] [configure] 1500 configuration steps out of 1840 steps complete.
2024-10-16 09:40:29 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-10-16 09:40:29 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-10-16 09:40:59 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-10-16 09:40:59 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-10-16 09:41:34 [NOTICE] [configure] 1750 configuration steps out of 1840 steps complete.
2024-10-16 09:41:55 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Exec resource 'foreman-rake-db:migrate' failed. Logs:
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]
    Adding autorequire relationship with User[foreman]
    Starting to evaluate the resource (1573 of 1840)
    Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
    '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
    Evaluated in 56.18 seconds
  Exec[foreman-rake-db:migrate](provider=posix)
    Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
    Executing '/usr/sbin/foreman-rake db:migrate'
    Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
    Executing '/usr/sbin/foreman-rake db:migrate'
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless
    Run `bin/rails db:migrate` to update your database then try again.
    You have 7 pending migrations:
      20230719080900 FixUbuntuVersions
      20231115151349 AddClonedFromToTemplates
      20240522093412 AddSmartProxyIdToTemplateInvocation
      20240522093413 MigrateSmartProxyIdsToTemplateInvocations
      20240522165308 AddPriorityToContentViewEnvironmentContentFacet
      20240621121212 KatelloRepositoryDebsIdBigint
      20240729192228 AddConvert2rhelToHostFacets
    Run `bin/rails db:migrate` to update your database then try again.
    You have 4 pending migrations:
      20240522093413 MigrateSmartProxyIdsToTemplateInvocations
      20240522165308 AddPriorityToContentViewEnvironmentContentFacet
      20240621121212 KatelloRepositoryDebsIdBigint
      20240729192228 AddConvert2rhelToHostFacets
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns
    rake aborted!
    StandardError: An error has occurred, this and all later migrations canceled:
        PG::ForeignKeyViolation: ERROR:  insert or update on table "template_invocations" violates foreign key constraint "fk_rails_4512fa74b9"
    DETAIL:  Key (smart_proxy_id)=(2) is not present in table "smart_proxies".
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:72:in `exec_delete'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:179:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:466:in `update_all'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:9:in `block (2 levels) in up'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `each'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `block in up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `loop'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:6:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `public_send'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `exec_migration'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:850:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:849:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1037:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `migrate_without_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1397:in `with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1086:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1061:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
        Caused by:
    ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  insert or update on table "template_invocations" violates foreign key constraint "fk_rails_4512fa74b9"
    DETAIL:  Key (smart_proxy_id)=(2) is not present in table "smart_proxies".
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:72:in `exec_delete'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:179:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:466:in `update_all'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:9:in `block (2 levels) in up'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `each'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `block in up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `loop'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:6:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `public_send'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `exec_migration'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:850:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:849:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1037:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `migrate_without_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1397:in `with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1086:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1061:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
        Caused by:
    PG::ForeignKeyViolation: ERROR:  insert or update on table "template_invocations" violates foreign key constraint "fk_rails_4512fa74b9"
    DETAIL:  Key (smart_proxy_id)=(2) is not present in table "smart_proxies".
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:72:in `exec_delete'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:179:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:466:in `update_all'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:9:in `block (2 levels) in up'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `each'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `block in up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `loop'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:6:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `public_send'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `exec_migration'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:850:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:849:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1037:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `migrate_without_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1397:in `with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1086:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1061:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    == 20230719080900 FixUbuntuVersions: migrating ================================
    == 20230719080900 FixUbuntuVersions: migrated (0.0080s) =======================
        == 20231115151349 AddClonedFromToTemplates: migrating =========================
    -- add_reference(:templates, :cloned_from, {:foreign_key=>{:to_table=>:templates, :on_delete=>:nullify}})
       -> 0.0225s
    == 20231115151349 AddClonedFromToTemplates: migrated (0.0225s) ================
        == 20240522093412 AddSmartProxyIdToTemplateInvocation: migrating ==============
    -- change_table(:template_invocations)
       -> 0.0084s
    == 20240522093412 AddSmartProxyIdToTemplateInvocation: migrated (0.0084s) =====
        == 20240522093413 MigrateSmartProxyIdsToTemplateInvocations: migrating ========
    change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
    rake aborted!
    StandardError: An error has occurred, this and all later migrations canceled:
        PG::ForeignKeyViolation: ERROR:  insert or update on table "template_invocations" violates foreign key constraint "fk_rails_4512fa74b9"
    DETAIL:  Key (smart_proxy_id)=(2) is not present in table "smart_proxies".
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:72:in `exec_delete'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:179:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:466:in `update_all'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:9:in `block (2 levels) in up'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `each'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `block in up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `loop'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:6:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `public_send'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `exec_migration'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:850:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:849:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1037:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `migrate_without_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1397:in `with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1086:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1061:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
        Caused by:
    ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  insert or update on table "template_invocations" violates foreign key constraint "fk_rails_4512fa74b9"
    DETAIL:  Key (smart_proxy_id)=(2) is not present in table "smart_proxies".
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:72:in `exec_delete'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:179:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:466:in `update_all'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:9:in `block (2 levels) in up'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `each'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `block in up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `loop'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:6:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `public_send'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `exec_migration'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:850:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:849:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1037:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `migrate_without_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1397:in `with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1086:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1061:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
        Caused by:
    PG::ForeignKeyViolation: ERROR:  insert or update on table "template_invocations" violates foreign key constraint "fk_rails_4512fa74b9"
    DETAIL:  Key (smart_proxy_id)=(2) is not present in table "smart_proxies".
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:72:in `exec_delete'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:179:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:466:in `update_all'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:9:in `block (2 levels) in up'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `each'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:7:in `block in up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `loop'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:229:in `in_batches'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'
    /usr/share/gems/gems/foreman_remote_execution-13.2.5/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb:6:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `public_send'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:870:in `exec_migration'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:850:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:849:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1037:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1380:in `ddl_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1302:in `migrate_without_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `block in migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1397:in `with_advisory_lock'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1251:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1086:in `up'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/migration.rb:1061:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `each'
    /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    == 20240522093413 MigrateSmartProxyIdsToTemplateInvocations: migrating ========

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/katello.log
4

May I assume you had at some point an proxy (with id 2) and that is gone now?

The error is in this migration: foreman_remote_execution/db/migrate/20240522093413_migrate_smart_proxy_ids_to_template_invocations.rb at master · theforeman/foreman_remote_execution · GitHub

cc @aruzicka – does that one not cope well with gone proxies?

5

Yes, I added a Smart Proxy awhile back and later removed it. Is it still lingering somewhere in the database and/or filesystem that I need to delete?

6

Huh, interesting, apparently I was expecting the links would be removed when the proxies are, but that’s not a correct assumption. This could do Fixes #37493 - Handle case where proxy no longer exists by adamruzicka · Pull Request #922 · theforeman/foreman_remote_execution · GitHub

If you want to just clean your db and go about your day, running the following in foreman-rake console should clean up the stale records

ForemanTasks::Link.joins(:task).where(resource_type: 'SmartProxy', task: { label: 'Actions::RemoteExecution::RunHostJob' }).where.not(resource_id: SmartProxy.select(:id)).destroy_all
2 Likes
7

That worked. Upgrade was successful. Thanks!

8

Thanks a lot !
Had the same issue (old smart-proxy removed but lots of stale records in db).
The command cleaned a bunch of old tasks and migration was successful after that!

9

Same error happened with me and your solution fixed it!!! Thank you so much.