Am I doing backup and restore wrong? Restored backup with hosts only, no organizations, content or settings

Support
1

Problem:
Restored backup with hosts only, no organizations, content and settings

I want to migrate current 3.2.1 Foreman from Centos 7 to Centos Stream 8 via fresh installation and restoring backup and then update it to supported version. After restoring the backup I found that I only have hosts in new installation.
Also backup didn’t compress pgsql_data.tar so i have to do it myself.

[root@Centos7Machine ~]# foreman-maintain backup offline --preserve-directory --features all /mnt/backup/

[root@Centos7Machine ~]# ls /mnt/backup/
config_files.tar.gz  metadata.yml  pgsql_data.tar  pgsql_data.tar.gz  pulp_data.tar

Expected outcome:
Full copy of previous state

Foreman and Proxy versions:
3.2.1

Other relevant data:

Is it normal behaviour? What have i done wrong?

2

Compressing pulp_data is not really useful, as it contains RPMs which already are compressed.

Did you uncompress it on the EL8 machine? As the restore process can’t do anything with the compressed version.

3

I don’t know what to do with pulp_data. Expected restore command would use it

[root@Centos8Machine ~]# foreman-maintain restore /mnt/backup 
[root@Centos8Machine ~]# runuser -u postgres -- reindexdb -a

I followed this guide Upgrading and Updating Foreman and this Administering Foreman

So I expected I need 3 commands:
foreman-maintain backup
foreman-maintain restore
unuser -u postgres – reindexdb -a

4

Can you show me an ls -alhZ /mnt/backup on Centos8Machine please?

5

Yeah, sure

[root@Centos8Machine ~]# ls -alhZ /mnt/backup
total 208G
drwxrwxrwx+ 1 32768    32772 system_u:object_r:nfs_t:s0  310 Mar  6 13:28 .
drwxr-xr-x. 3 root  root     system_u:object_r:mnt_t:s0   20 Mar  5 17:59 ..
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0 1.8G Mar  5 13:50 config_files.tar.gz
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0  70K Mar  5 13:50 .config.snar
drwxrwx---+ 1 root  postgres system_u:object_r:nfs_t:s0  194 Mar  6 15:12 katello-backup-2024-03-06-13-28-24
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0  45K Mar  5 13:48 metadata.yml
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0   45 Mar  5 16:55 pgsql_data.tar
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0  177 Mar  5 16:56 pgsql_data.tar.gz
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0  36K Mar  5 15:36 .postgres.snar
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0 207G Mar  5 15:35 pulp_data.tar
-rw-rw-rw-+ 1 root  root     system_u:object_r:nfs_t:s0 6.1M Mar  5 15:35 .pulp.snar
6

Wait a moment. That pgsql_data is way too small.

Can you show me the full output when you run f-m backup offline on the el7 box? There is something wrong there.

7

We have 300 registered hosts so maybe the size is ok?

I dont know where to find previous backup output. So I will show the recent one.

foreman-maintain backup offline /mnt/backup/ 
[root@Centos7Machine ~]# foreman-maintain backup offline  /mnt/backup/
Starting backup: 2024-03-06 13:28:24 +0300
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up: 
/ Checking connection to the Foreman DB                               [OK]      
--------------------------------------------------------------------------------


Running Backup
================================================================================
Confirm turning off services is allowed: 
WARNING: This script will stop your services.

Do you want to proceed?, [y(yes), q(quit)] y
                                                                      [OK]      
--------------------------------------------------------------------------------
Prepare backup Directory: 
Creating backup folder /mnt/backup/katello-backup-2024-03-06-13-28-24 [OK]
--------------------------------------------------------------------------------
Check if the directory exists and is writable:                        [OK]
--------------------------------------------------------------------------------
Generate metadata: 
- Saving metadata to metadata.yml                                     [OK]      
--------------------------------------------------------------------------------
Detect features available in the local proxy:                         [OK]
--------------------------------------------------------------------------------
disable active sync plans: 
- Total 0 sync plans are now disabled.                                [OK]      
--------------------------------------------------------------------------------
Add maintenance_mode tables/chain to nftables/iptables:               [OK]
--------------------------------------------------------------------------------
Stop applicable services: 

Stopping the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-api.socket, pulpcore-content.socket, pulpcore-worker@1.service, pulpcore-worker@2.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
/ stopping pulpcore-content.socket                                              
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.
\ All services stopped                                                [OK]      
--------------------------------------------------------------------------------
Backup config files: 
/ Collecting config files to backup                                   [OK]      
--------------------------------------------------------------------------------
Backup Pulp data: 
| Collecting Pulp data                                                [OK]      
--------------------------------------------------------------------------------
Backup Candlepin DB offline: 
- Collecting data from /var/opt/rh/rh-postgresql12/lib/pgsql/data/    [OK]      
--------------------------------------------------------------------------------
Backup Foreman DB offline: 
Already done                                                          [OK]
--------------------------------------------------------------------------------
Backup Pulpcore DB offline: 
Already done                                                          [OK]
--------------------------------------------------------------------------------
Start applicable services: 

Starting the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
\ starting pulpcore-content                                                     
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.
| All services started                                                [OK]      
--------------------------------------------------------------------------------
re-enable sync plans: 
/ Total 1 sync plans are now enabled.                                 [OK]      
--------------------------------------------------------------------------------
Remove maintenance mode table/chain from nftables/iptables:           [OK]
--------------------------------------------------------------------------------
Compress backup data to save space: 
- Compressing backup of Postgres DB                                   [OK]      
--------------------------------------------------------------------------------

Done with backup: 2024-03-06 15:12:14 +0300
**** BACKUP Complete, contents can be found in: /mnt/backup/katello-backup-2024-03-06-13-28-24 ****
ls -lha /mnt/backup/katello-backup-2024-03-06-13-28-24/ 
[root@Centos7Machine ~]# ls -lha /mnt/backup/katello-backup-2024-03-06-13-28-24/
total 210G
drwxrwx---+ 1 root  postgres  194 Mar  6 15:12 .
drwxrwxrwx+ 1 32768    32772  310 Mar  6 13:28 ..
-rw-rw-rw-+ 1 root  root     1.8G Mar  6 13:31 config_files.tar.gz
-rw-rw-rw-+ 1 root  root      70K Mar  6 13:31 .config.snar
-rw-rw-rw-+ 1 root  root      45K Mar  6 13:29 metadata.yml
-rw-rw-rw-+ 1 root  root     1.9G Mar  6 15:05 pgsql_data.tar.gz
-rw-rw-rw-+ 1 root  root      36K Mar  6 15:05 .postgres.snar
-rw-rw-rw-+ 1 root  root     207G Mar  6 15:04 pulp_data.tar
-rw-rw-rw-+ 1 root  root     6.1M Mar  6 15:04 .pulp.snar
ls -alhZ /mnt/backup/katello-backup-2024-03-06-13-28-24/ 
[root@Centos7Machine ~]# ls -alhZ /mnt/backup/katello-backup-2024-03-06-13-28-24/
drwxrwx---+ root  postgres ?                                .
drwxrwxrwx+ 32768    32772 ?                                ..
-rw-rw-rw-+ root  root     ?                                config_files.tar.gz
-rw-rw-rw-+ root  root     ?                                .config.snar
-rw-rw-rw-+ root  root     ?                                metadata.yml
-rw-rw-rw-+ root  root     ?                                pgsql_data.tar.gz
-rw-rw-rw-+ root  root     ?                                .postgres.snar
-rw-rw-rw-+ root  root     ?                                pulp_data.tar
-rw-rw-rw-+ root  root     ?                                .pulp.snar

I will try to restore recent backup (without features all option) this time

8

So, when restoring, you need to call f-m backup restore /mnt/backup/katello-backup-2024-03-06-13-28-24/ not /mnt/backup/

(the pgsql_data.tar.gz in the original paste was only 177 bytes big, that’s not enough)