How is foreman-maintain restore -i supposed to work?

Development
#1

Hi,

I am putting this here instead of Support-section, because if I am correct then code-/documentation-changes have to be made :innocent: .

I played around with foreman-maintain backup/restore under foreman-3.3 and especially in combination with --incremental I spotted some odd things.

If I create incremental offline backups, incremental tar-archives of /var/lib/pgsql are created.

If I want to restore them now, the documentation says (supposedly being pretty outdated at version 0.2) to restore the full-backup and all consecutive incremental-backups in separate foreman-maintain restore commands like so:

foreman-maintain restore /tmp/backups/FULL_BACKUP
foreman-maintain restore -i /tmp/backups/FIRST_INCREMENTAL
foreman-maintain restore -i /tmp/backups/SECOND_INCREMENTAL

The problem now is that the restore commands does more than just extract the archives. It does

$ foreman-maintain restore /tmp/backups/FULL_BACKUP
[Stuff before FULL_BACKUP restore]
* extract FULL_BACKUP to /var/lib/pgsql/data
* Start postgresql.service
* DB: REINDEX
* ...

$ foreman-maintain restore -i /tmp/backups/FIRST_INCREMENTAL
[Stuff before FIRST_INCREMENTAL restore]
* extract FIRST_INCREMENTAL to /var/lib/pgsql/data
...

Which is effectively

  1. restore the db-data-dir
  2. make changes to the DB
  3. restore the db-data-dir from FIRST_INCREMENTAL
  4. make changes to the DB
    …

Which practically has to fail at step 4, does it not? Because the db-data-dir pretty sure changed due to starting the postgresql.service and running operations on the DB.

Since the metadata.yaml included in the backups kinda points to the previous incremental/full backup, I would expect foreman-maintain to detect this and do something like this:

$ foreman-maintain restore -i /tmp/backups/SECOND_INCREMENTAL
[do stuff before restore]
-> extract files from SECOND_INCREMENTAL/*.tar.gz
-> extract files from FIRST_INCREMENTAL/*.tar.gz
-> extract files from FULL_BACKUP/*.tar.gz
* Start postgresql.service
* DB: REINDEX
* ...

Am I missing something?

Another option would be to enforce inclusion of db-dump files (though this would always backup the full DB).

#2

Hi Markus,

about documentation: Please have a look at Performing an Incremental Backup and Restoring from Incremental Backups in Administering Foreman+Katello. I’ve linked nightly, you can also choose other versions such as 3.3.

I highly suspect that the docs for foreman-maintain are not up to date: last change: 4 years ago on Github.

#3

Do you have an idea @ehelms ?

#4

I’m looking into this.

#5

Okay, so a few things:

  1. the way you’re calling it is correct (the -i is not needed, it gets autodetected, but also doesn’t hurt)
  2. yeah, I can reproduce the fact that restoring the incremental is broken
  3. not sure it’s really the “reindex” step that breaks it (we had other db altering steps in the past too), testing that now
#6

no, you’re right, it seems to be the reindex that fucks things up :thinking:

#7

I’ve opened Bug #36668: restore of offline incremental backups is broken - Foreman Maintain - Foreman to properly track this.

1 Like