Restore of Foreman configuration fails after upgrade to 3.11

Support
1

Problem:
Testing of the restore process after upgrading to Foreman 3.11

As part of my testing of Foreman after upgrading, I want to execute a standard restore after creating a backup using the command ‘foreman-maintain restore /backup/katello-backup-2024-09-19-18-10-05’

Here is the output from the restore attempt

Running preparation steps required to run the next scenarios
================================================================================
Make sure Candlepin DB is up:
/ Checking connection to the Candlepin DB                             [OK]
--------------------------------------------------------------------------------
Make sure Foreman DB is up:
- Checking connection to the Foreman DB                               [OK]
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up:
\ Checking connection to the Pulpcore DB                              [OK]
--------------------------------------------------------------------------------


Running Restore backup
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Validate backup has appropriate files:                                [OK]
--------------------------------------------------------------------------------
Validate hostname is the same as backup:                              [OK]
--------------------------------------------------------------------------------
Validate network interfaces match the backup:                         [OK]
--------------------------------------------------------------------------------
Validate permissions for PostgreSQL dumps:                            [OK]
--------------------------------------------------------------------------------
Confirm dropping databases and running restore:

WARNING: This script will drop and restore your database.
Your existing installation will be replaced with the backup database.
Once this operation is complete there is no going back.
Do you want to proceed?, [y(yes), q(quit)] y
                                                                      [OK]
--------------------------------------------------------------------------------
Ensure required packages are installed before restore:
/ Installing required packages                                        [OK]
--------------------------------------------------------------------------------
Restore configs from backup:
- Restoring configs                                                   [OK]
--------------------------------------------------------------------------------
Run installer reset:
/ Installer reset                                                     [FAIL]
Failed executing yes | foreman-installer -v --reset-data , exit status 1:
 2024-09-20 09:11:10 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-09-20 09:11:12 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-09-20 09:11:12 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Are you sure you want to continue? This will drop the databases, reset all configurations that you have made and bring all application data back to a fresh install. [y/n]
2024-09-20 09:11:17 [NOTICE] [pre] Dropping foreman database!
2024-09-20 09:11:17 [ERROR ] [root] PGPASSWORD='Smile067' psql -U foreman -h foremandb -p 5432 -d foreman -t -c "DROP OWNED BY CURRENT_USER CASCADE;" failed! Check the output for error!
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.

The following steps ended up in failing state:

  [restore-installer-reset]

Resolve the failed steps and rerun the command.
In case the failures are false positives, use
--whitelist="restore-installer-reset"



Running Rescue Restore backup
================================================================================

Expected outcome:

Successful restore of Foreman backup to server

Foreman and Proxy versions:

Foreman version 3.11.2

Foreman and Proxy plugin versions:

|foreman-tasks|9.1.1|
|foreman_ansible|14.0.0|
|foreman_dhcp_browser|0.0.8|
|foreman_puppet|7.0.0|
|foreman_remote_execution|13.1.0|
|katello|4.13.1|

Distribution and version:
AlmaLinux 8.10

Other relevant data:
Attempt to run the postgresql drop manually to get additional troubleshooting info

# PGPASSWORD='XXXXXX' psql -U foreman -h foremandb -p 5432 -d foreman -t -c "DROP OWNED BY CURRENT_USER CASCADE;"
ERROR:  cannot drop type evr_t because extension evr requires it
HINT:  You can drop extension evr instead.
2

You’re using an external database, right?

It’s a bug in the PostgreSQL upgrade process (see PostgreSQL: BUG #18625: user-created extensions change ownership to "postgres" after upgrade for details).

The easiest is probably to execute something like this on your external DB server on the foreman database as the superuser (untested, sorry):

UPDATE pg_extension SET extowner=(SELECT oid FROM pg_roles where rolname='foreman') where extname='evr';
3

Yes, it is an external database that we upgrade to postgresql 13 from 12 as part of the Foreman upgrade.

We updated the ownership of the evr extension to foreman per your suggested. That did not resolve the issue. Our DBA did the following and now the restore appears to proceeding.

foreman=# drop extension evr cascade;
NOTICE:  drop cascades to 6 other objects
DETAIL:  drop cascades to trigger evr_insert_trigger_katello_installed_packages on table katello_installed_packages
drop cascades to trigger evr_insert_trigger_katello_rpms on table katello_rpms
drop cascades to trigger evr_update_trigger_katello_installed_packages on table katello_installed_packages
drop cascades to trigger evr_update_trigger_katello_rpms on table katello_rpms
drop cascades to column evr of table katello_installed_packages
drop cascades to column evr of table katello_rpms
DROP EXTENSION

Is there something else that we should look into concerning this that needs further investigation?

4

Nah, that seems fine.

Should be a one-time thing too – as the newly created evr extension (during the restore) will now have proper ownership again.

2 Likes
5

Second run through the backup/restore process executed as expected. So that confirms that this is a one off issue that can occur when an upgrade to PostgreSQL is part of the upgrade process.