Problem backing up with foreman-maintain

OS: CentOS 7.8
Foreman: 1.24.2
Katello: 3.14.1

I attempted to backup my system with foreman-maintain. The command I used was this:

foreman-maintain backup offline --whitelist=“sync-plans-disable” --skip-pulp-content -y /var/backup

It produced this output:

[root@fm ~]# foreman-maintain backup offline --whitelist=“sync-plans-disable” --skip-pulp-content -y /var/backup
Starting backup: 2021-02-02 11:09:15 -0500
Running preparation steps required to run the next scenarios

Make sure Foreman DB is up:
/ Checking connection to the Foreman DB [OK]

Running Backup

Confirm turning off services is allowed:
WARNING: This script will stop your services.

                                                                  [OK]

Prepare backup Directory:
Creating backup folder /var/backup/katello-backup-2021-02-02-11-09-15 [OK]

Check if the directory exists and is writable: [OK]

Generate metadata:
\ Saving metadata to metadata.yml [OK]

Detect features available in the local proxy: [OK]

disable active sync plans: [SKIPPED]

Add maintenance_mode chain to iptables: [OK]

Stop applicable services:
Stopping the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, goferd, httpd, pup
petserver, foreman-proxy
\ All services stopped [OK]

Backup config files:
\ Collecting config files to backup [OK]

Backup Pulp data: [SKIPPED]

Backup mongo offline:

  • Collecting Mongo data [OK]

Backup Candlepin DB offline:
/ Collecting data from /var/lib/pgsql/data/ [OK]

Backup Foreman DB offline:
Already done [OK]

Start applicable services:
Starting the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, goferd, httpd, puppetserver, foreman-proxy
| starting puppetserver
Job for puppetserver.service failed because the control process exited with error code. See “systemctl status puppetserver.service” and “journalctl -xe” for details.
\ All services started [OK]

re-enable sync plans:
/ re-enabling sync plans [FAIL]
Could not update the sync plan:
Couldn’t find sync plan ‘4’ in organization ‘’

Scenario [Backup] failed.

The following steps ended up in failing state:

[sync-plans-enable]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“sync-plans-enable”

Running Failed backup cleanup

Start applicable services:
Starting the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, goferd, httpd, puppetserver, foreman-proxy

  • starting puppetserver
    Job for puppetserver.service failed because the control process exited with error code. See “systemctl status puppetserver.service” and “journalctl -xe” for details.
    \ All services started [OK]

re-enable sync plans:
\ re-enabling sync plans [FAIL]
Could not update the sync plan:
Couldn’t find sync plan ‘4’ in organization ‘’

Remove maintenance_mode chain from iptables: [OK]

Clean up backup directory: [OK]

Scenario [Failed backup cleanup] failed.

The following steps ended up in failing state:

[sync-plans-enable]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“sync-plans-enable”

Done with backup: 2021-02-02 11:25:12 -0500
Backup didn’t finish. Incomplete backup was removed.

It was doing great until it got to the point of re-enabling the syncplans. Then it cleared out the data it backed up and failed.

I looked at the file /var/lib/foreman-maintain/data.yml and noticed that it had 3 sync plans listed by number that were long since deleted from the system so I removed them from that file and re-ran the backup. It still failed at the same point. This time it just said:

Couldn’t find sync plan ‘’ in organization ‘’

I looked up the error and some one suggested removing the data.yml file and then running foreman-maintain upgrade check --target-version= but I do not know what version to look for. So I tried foreman-maintain upgrade list-versions and it was unable to list any versions.

Anyone have any suggestions?

Hi @Louis_Bohm, thanks for letting us know, can you try changing to ---whitelist=" --whitelist=“sync-plans-disable,sync-plans-enable”, it may be that you have to disable both.

What version of foreman-maintain is this? sudo rpm -q rubygem-foreman_maintain - I know there was an issue with skipping steps that was fixed a long time ago Fixes #22244 - Skip whitelisted steps by swapab · Pull Request #153 · theforeman/foreman_maintain · GitHub

I’m actually not sure why those steps run on an offline backup since we shutdown services and nothing would sync during the backup (it’s more for online backups), but I can look into that aspect more.

Well this is odd. When I look I do not see rubygem-foreman_maintain installed. Where did foreman-maintain come from then???

I am going to install rubygem-foreman_maintain and try to run it normally and if that fails I will try with your suggestions.

I probably cannot try an offline backup until latter in the week.

Louis
-<<—->>-
Louis Bohm
louisbohm@gmail.com

1 Like

I was able to test this again and it failed.

foreman-maintain version:
[root@fm ~]# rpm -q rubygem-foreman_maintain
rubygem-foreman_maintain-0.5.4-1.el7.noarch

Output of the command:
[root@fm ~]# foreman-maintain backup offline --skip-pulp-content -y /var/backup
Starting backup: 2021-02-08 15:11:29 -0500
Running preparation steps required to run the next scenarios

Here is an attempt to backup using rubygem-foreman_maintain-0.5.4-1.el7.noarch.

[root@fm ~]# foreman-maintain backup offline --skip-pulp-content -y /var/backup
Starting backup: 2021-02-18 08:41:05 -0500
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up:
/ Checking connection to the Foreman DB                               [OK]
--------------------------------------------------------------------------------


Running Backup
================================================================================
Confirm turning off services is allowed:
WARNING: This script will stop your services.

Do you want to proceed? (assuming yes)
                                                                      [OK]
--------------------------------------------------------------------------------
Prepare backup Directory:
Creating backup folder /var/backup/foreman-backup-2021-02-18-08-41-05 [OK]
--------------------------------------------------------------------------------
Check if the directory exists and is writable:                        [OK]
--------------------------------------------------------------------------------
Generate metadata:
- Saving metadata to metadata.yml                                     [OK]
--------------------------------------------------------------------------------
Detect features available in the local proxy:                         [OK]
--------------------------------------------------------------------------------
Add maintenance_mode chain to iptables:                               [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, goferd, httpd, puppetserver, foreman-proxy
\ All services stopped                                                [OK]
--------------------------------------------------------------------------------
Backup config files:
/ Collecting config files to backup                                   [FAIL]
undefined method `answers' for nil:NilClass
--------------------------------------------------------------------------------
Scenario [Backup] failed.

The following steps ended up in failing state:

  [backup-config-files]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="backup-config-files"



Running Failed backup cleanup
================================================================================
Start applicable services:
Starting the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, goferd, httpd, puppetserver, foreman-proxy
/ All services started                                                [OK]
--------------------------------------------------------------------------------
Remove maintenance_mode chain from iptables:                          [OK]
--------------------------------------------------------------------------------
Clean up backup directory:                                            [OK]
--------------------------------------------------------------------------------

Done with backup: 2021-02-18 08:49:31 -0500
Backup didn't finish. Incomplete backup was removed.

Why is it not able to backup the backup-config-files?

This command finally let me get a backup: foreman-maintain backup offline --whitelist=“backup-config-files” --skip-pulp-content -y /var/backup. Not sure what I am now missing that was part of the backup-config-files. I can only guess its the yaml files for puppet.