Problem:
Backup of Foreman/Katello environment via ‘foreman-maintain backup offline -y --skip-pulp-content /backup’ where the Foreman postgres DB is on a remote DB server.
Produced the following results:
[root@foreman foreman]# foreman-maintain backup offline -y --skip-pulp-content /backup
Starting backup: 2021-02-18 16:08:44 -0600
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up:
/ Checking connection to the Foreman DB [OK]
--------------------------------------------------------------------------------
Running Backup
================================================================================
Confirm turning off services is allowed:
WARNING: This script will stop your services.
Do you want to proceed? (assuming yes)
[OK]
--------------------------------------------------------------------------------
Prepare backup Directory:
Creating backup folder /backup/katello-backup-2021-02-18-16-08-44 [OK]
--------------------------------------------------------------------------------
Check if the directory exists and is writable: [OK]
--------------------------------------------------------------------------------
Generate metadata:
| Saving metadata to metadata.yml [OK]
--------------------------------------------------------------------------------
Detect features available in the local proxy: [OK]
--------------------------------------------------------------------------------
disable active sync plans:
/ Total 6 sync plans are now disabled. [OK]
--------------------------------------------------------------------------------
Add maintenance_mode chain to iptables: [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):
rh-mongodb34-mongod, rh-redis5-redis, postgresql, postgresql (foreman), pulpcore-api, pulpcore-api.socket, pulpcore-content, pulpcore-content.socket, pulpcore-resource-manager, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, pulpcore-worker@*, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, foreman.socket, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
/ stopping postgresql (foreman)
postgresql (foreman) is remote and is UP.
| All services stopped [OK]
--------------------------------------------------------------------------------
Backup config files:
- Collecting config files to backup [OK]
--------------------------------------------------------------------------------
Backup Pulp data: [SKIPPED]
--------------------------------------------------------------------------------
Backup mongo offline:
- Collecting Mongo data [OK]
--------------------------------------------------------------------------------
Backup Candlepin DB offline:
\ Collecting data from /var/opt/rh/rh-postgresql12/lib/pgsql/data/ [OK]
--------------------------------------------------------------------------------
Backup Foreman DB offline:
Backup of /var/opt/rh/rh-postgresql12/lib/pgsql/data/ is not supported for remote databases. Doing postgres dump instead...
/ Getting Foreman DB dump [OK]
--------------------------------------------------------------------------------
Backup Pulpcore DB offline:
Already done [OK]
--------------------------------------------------------------------------------
Start applicable services:
Starting the following service(s):
rh-mongodb34-mongod, rh-redis5-redis, postgresql, postgresql (foreman), pulpcore-api, pulpcore-content, pulpcore-resource-manager, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, pulpcore-worker@*, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
| starting postgresql (foreman)
postgresql (foreman) is remote and is UP.
| All services started [OK]
--------------------------------------------------------------------------------
re-enable sync plans:
| Total 6 sync plans are now enabled. [OK]
--------------------------------------------------------------------------------
Remove maintenance_mode chain from iptables: [OK]
--------------------------------------------------------------------------------
Compress backup data to save space:
- Compressing backup of Postgres DB
| Compressing backup of Mongo DB [OK]
--------------------------------------------------------------------------------
Done with backup: 2021-02-18 16:15:59 -0600
**** BACKUP Complete, contents can be found in: /backup/katello-backup-2021-02-18-16-08-44
****
When attempt to validate backup via execution of ‘foreman-maintain restore /backup/katello-backup-2021-02-18-16-08-44/’, the results were as follows:
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up:
/ Checking connection to the Foreman DB [OK]
--------------------------------------------------------------------------------
Running Restore backup
================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Validate backup has appropriate files: [FAIL]
The given directory does not contain the required files or has too many files
All backup directories contain: config_files.tar.gz
An online or remote database backup directory contains: mongo_dump, candlepin.dump, foreman.dump, pulpcore.dump
An offline backup directory contains: mongo_data.tar.gz, pgsql_data.tar.gz
A logical backup directory contains: mongo_dump, candlepin.dump, foreman.dump, pulpcore.dump, mongo_data.tar.gz, pgsql_data.tar.gz
Including pulp_data.tar is optional and will restore pulp data to the filesystem if included.
Only the following files were found: mongo_data.tar.gz, pgsql_data.tar.gz, foreman.dump, config_files.tar.gz, metadata.yml
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.
The following steps ended up in failing state:
[restore-validate-backup]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="restore-validate-backup"
Running Resuce Restore backup
================================================================================
Any suggestions on how to successfully restore in this scenerio?
Expected outcome:
I expected the restore to accept the available files in the backup directory to be a valid set for restoring Foreman.
Here is the list of files in backup folder:
[root@foreman katello-backup-2021-02-18-16-08-44]# ll
total 2330288
-rw-rw-r–+ 1 root root 1920255 Feb 18 16:09 config_files.tar.gz
-rw-rw-r–+ 1 root root 28870135 Feb 18 16:09 foreman.dump
-rw-rw-r–+ 1 root root 44427 Feb 18 16:09 metadata.yml
-rw-rw-r–+ 1 root root 560521 Feb 18 16:09 mongo_data.tar.gz
-rw-rw-r–+ 1 root root 2354813243 Feb 18 16:09 pgsql_data.tar.gz
Foreman and Proxy versions:
Foreman version 2.3.2
Foreman and Proxy plugin versions:
foreman-tasks 3.0.3
foreman_remote_execution 4.2.2
katello 3.18.1
Distribution and version:
CentOS 7.9.2009
Other relevant data:
Command to backup Foreman
foreman-maintain backup offline -y --skip-pulp-content /backup
Command to restore Foreman
foreman-maintain restore /backup/katello-backup-2021-02-18-16-08-44/