Missing pgsql data required for restore procedure

Problem: pgsql_data.tar.gz is missing when doing a full-backup

Expected outcome:
pgsql_data.tar.gz exists

Foreman and Proxy versions:
foreman 3.12/ katello 4.14

Distribution and version:
RH8

Other relevant data:
Everything runs on one VM (foreman,proxy,katello,databases,puppet) and is supposed to be migrated to an RH9 VM due to a failed leapp upgrade (manual restore)

What I did is just the default backup procedure

foreman-maintain backup offline /storage/FM_backup/

When I tried to restore it says that there is pgsql_data.tar.gz missing which wasnt generated in the backup. procedure. All files generated during successfully run backup can be seen in the pic below.
RESTORE:

Whats the correct procedure here?

You have the *.dump files available, so it should be able to restore that.

But I think what it’s complaining about are the candlepin.dump, pulpcore.dump and pulp_data.tar files which it considers “too many”.

Does that system have a working, empty Katello installation right now? As that’s the requirement for a restore – but for some reason the list of files it lists are for a plain Foreman installation.

Indeed I missed to run the katello scenario with foreman-installer why it didnt work. Now it worked, thank you! However, I stumpled upon several other problems:

  1. Complain about permissions “postgres user needs read access to the following files” even though if I change to postgres its no problem to open them as postgres → solution as recommended by script:
foreman-maintain restore . --whitelist="restore-validate-postgresql-dump-permissions"
  1. /tmp FS must not be mounted with noexec! Otherwise it shows:
2025-05-09 11:38:33 [NOTICE] [configure] 1250 configuration steps out of 1866 steps complete.
2025-05-09 11:38:41 [ERROR ] [configure] Systemd start for puppetserver failed!
2025-05-09 11:38:41 [ERROR ] [configure] journalctl log for puppetserver:
2025-05-09 11:38:41 [ERROR ] [configure] May 09 11:33:47 foreman-test puppetserver[3836809]: Failed to load feature test for posix: can't find user for 0
2025-05-09 11:38:41 [ERROR ] [configure] May 09 11:33:47 foreman-test puppetserver[3836809]: Execution error (RuntimeError) at RUBY/<main> (/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/feature/base.rb:21).

And now it fails when importing the candlepin.dump:

foreman-maintain restore . --whitelist="restore-validate-postgresql-dump-permissions"
Running preparation steps required to run the next scenarios
================================================================================
Make sure Candlepin DB is up:
/ Checking connection to the Candlepin DB                             [OK]
--------------------------------------------------------------------------------
Make sure Foreman DB is up:
- Checking connection to the Foreman DB                               [OK]
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up:
\ Checking connection to the Pulpcore DB                              [OK]
--------------------------------------------------------------------------------


Running Restore backup
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Validate backup has appropriate files:                                [OK]
--------------------------------------------------------------------------------
Validate hostname is the same as backup:                              [OK]
--------------------------------------------------------------------------------
Validate network interfaces match the backup:                         [OK]
--------------------------------------------------------------------------------
Validate permissions for PostgreSQL dumps:                            [SKIPPED]
--------------------------------------------------------------------------------
Confirm dropping databases and running restore:

WARNING: This script will drop and restore your database.
Your existing installation will be replaced with the backup database.
Once this operation is complete there is no going back.
Do you want to proceed?, [y(yes), q(quit)] yes
                                                                      [OK]
--------------------------------------------------------------------------------
Ensure required packages are installed before restore:
/ Installing required packages                                        [OK]
--------------------------------------------------------------------------------
Restore configs from backup:
- Restoring configs                                                   [OK]
--------------------------------------------------------------------------------
Stop cron service:

Stopping the following service(s):
crond
/ All services stopped                                                [OK]
--------------------------------------------------------------------------------
Run installer reset:
| Installer reset                                                     [OK]
--------------------------------------------------------------------------------
Stop applicable services:

Stopping the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, tomcat, pulpcore-api.socket, pulpcore-content.socket, dynflow-sidekiq@orchestrator, foreman, httpd, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
\ All services stopped                                                [OK]
--------------------------------------------------------------------------------
Extract any existing tar files in backup:
| Extracting pulp data                                                [OK]
--------------------------------------------------------------------------------
Drop postgresql databases:
/ Dropping pulpcore database                                          [OK]
--------------------------------------------------------------------------------
Start applicable services:

Starting the following service(s):
postgresql
| All services started                                                [OK]
--------------------------------------------------------------------------------
Restore candlepin postgresql dump from backup:
/ Restoring candlepin dump                                            [FAIL]
Failed executing runuser - postgres -c 'pg_restore -C -d postgres ./candlepin.dump', exit status 1:
 pg_restore: error: could not open input file "./candlepin.dump": No such file or directory
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.

The following steps ended up in failing state:

  [restore-candlepin-dump]

Resolve the failed steps and rerun the command.
In case the failures are false positives, use
--whitelist="restore-candlepin-dump,restore-validate-postgresql-dump-permissions"



Running Rescue Restore backup
================================================================================

If I do it manually with user postgres it works:

postgres@foreman-test [Foreman-T1] /tmp/RESTORE/katello-backup-2025-05-08 $ pg_restore -C -d postgres ./candlepin.dump
postgres@foreman-test [Foreman-T1] /tmp/RESTORE/katello-backup-2025-05-08 $ echo $?
0

Afterwards a rerun of the restore wont work anymore because it cant start the foreman-db which is a prerequisite. → so again restore from snapshot
Can you help me resolve this issue?