Unable to take incremental backup with foreman-maintain service

FaristaS · January 24, 2019, 11:55am

**Problem:Unable to take incremental backup with foreman-maintain service

**Expected outcome:should able to take incremental backup with foreman-maintain service

**Foreman and Proxy versions:1.19

Foreman and Proxy plugin versions:

Hi All,
We have set up a script in cron to take offline backup of foreman using foreman-maintain service.
Every Sunday it takes full backup (including files,etc and pulp).
And for other day, it takes incremental backup as mentioned in the script, but instead of incremental it takes full backup.
We are running foreman version 1.19 and katello version 3.8 in our environment.
Below is the command available for incremental backup.
#foreman-maintain backup offline --incremental --assumeyes “$LAST” $DESTINATION
where LAST=$(ls -td – $DESTINATION/*/ | head -n 1)

Please provide your valuable inputs

John_Mitsch · January 24, 2019, 4:54pm

It looks like you aren’t passing the full backup directory to --incremental, the directory needs to come after this.
i.e.

foreman-maintain backup offline -y --incremental /backup/katello-backup-2019-01-24-16-48-24 /backup/

So you could do this:

foreman-maintain backup offline -y --incremental “$LAST” $DESTINATION

FaristaS · January 24, 2019, 5:17pm

Hi John,

Appreciated for quick reply.
Let me provide some more inputs from my side.
We have the below script for sunday full and other day incremental backup:
#########################################################
#!/bin/bash -e
export PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESTINATION=/backup
if [[ $(date +%w) == 0 ]]; then
foreman-maintain backup offline $DESTINATION
else
LAST=$(ls -td – $DESTINATION/*/ | head -n 1)
foreman-maintain backup offline --incremental “$LAST” $DESTINATION
fi
exit 0
##########################################################

So all we have is if statement that takes care of full backup and else statement that takes care of incremental backup.
The issue here is instead of incremental, it takes full backup other day as well.

“/backup/katello-backup-2019-01-24-16-48-24” is under the variable “$LAST” in this case.

Please correct me if i am missing something.

Thanks
Farista Singh

John_Mitsch · January 24, 2019, 5:45pm

I’m unable to reproduce this on a katello 3.8 box. Can you see what happens when you run the backup without the script? This could rule out any issues with the script itself. For example:

foreman-maintain backup offline --incremental /backup/katello-backup-2019-01-24-16-48-24 /backup

and compare the size of the files in the backup produced with the original

FaristaS · January 24, 2019, 5:52pm

Sure John,
I will update with the results soon.

Thanks
Farista Singh

FaristaS · January 25, 2019, 5:14am

Hi John,
Its really surprising to know that it runs perfectly on CLI, below is the incremental backup cmd:
#foreman-maintain backup offline --incremental /backup/katello-backup-2019-01-25-01-00-05 /backup/

But when it comes to setting up a cron and passing variables to $LAST_FOLDER and $BACKUP_DIR it doesn’t work. viz:
#foreman-maintain backup offline --assumeyes --incremental $LAST_FOLDER $BACKUP_DIR

where
LAST_FOLDER=$(ls -td – $DESTINATION/*/ | head -n 1)
BACKUP_DIR=/backup/

One more surprising thing is, when i run the script using sh on CLI, then also it WORKS FINE.
I have re-edited the cron and scheduled it, let’s wait for couple of days(cron runs once in a day) to get the required output.

Lets hope for the good.

Thanks
Farista Singh

John_Mitsch · January 25, 2019, 2:33pm

What version of foreman-maintain are you using? I know some issues running in a cron environment have been fixed

FaristaS · January 28, 2019, 4:50am

Hi John,
Thanks for the update.
we have rubygem-foreman_maintain-0.2.4-1.el7.noarch installed to get foreman-maintain command and it has the required addition of lines that you have mentioned.
Last week i did some modification in script, but no luck.
Still using cron, my script takes full backup everyday(it should take incremental)
while if I run the same command on CLI, it goes for incremental backup.

Not sure what could be the issue.
Below is our script running in our cron.
#######################################################################
#!/bin/bash -e
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESTINATION=/backup_directory
if [[ $(date +%w) == 0 ]]; then
foreman-maintain backup offline --assumeyes $DESTINATION
else
LAST=$(ls -td – $DESTINATION/*/ | head -n 1)
foreman-maintain backup offline --assumeyes --incremental $LAST $DESTINATION
fi
exit 0
#######################################################################

John_Mitsch · January 28, 2019, 2:26pm

@mbacovsky Do you have any ideas?

A quick recap:
@FaristaS is running this script from a cron job

#!/bin/bash -e
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESTINATION=/backup_directory
if [[ $(date +%w) == 0 ]]; then
foreman-maintain backup offline --assumeyes $DESTINATION
else
LAST=$(ls -td – $DESTINATION/*/ | head -n 1)
foreman-maintain backup offline --assumeyes --incremental $LAST $DESTINATION
fi
exit 0

The script runs fine, but it is taking a full backup every day instead of an incremental backup on weekdays. They can run the script using sh and it runs an incremental backup, so it appears to be only happening when triggered by cron.

@FaristaS feel free to add or correct this summary. My only other thought is to use an updated version of foreman-maintain -> https://yum.theforeman.org/latest/el7/x86_64/rubygem-foreman_maintain-0.3.0-1.el7.noarch.rpm

mbacovsky · January 28, 2019, 11:37pm

@FaristaS my only thought so far is if adding SHELL=/bin/bash to the crontab would make any difference?
Also /var/log/foreman-maintain/foreman-maintain.log may reveal some details. All params passed in as well as some internal statuses should be included there. Could you share the log of running from cron?

FaristaS · January 29, 2019, 4:55am

Hi John,

You got that perfect, I will have to check whether we can update foreman-maintain version (from 0.2 to 0.3).
One more thing I want to bring into notice is:
Daily it takes full backup, but when i run the same cron second time manually, it takes incremental.
strange but true. <— Let me know your inputs on this

We have set up cron once in a day, sunday full backup and other days incremental <-- This is what we want to achieve.

Thanks
Farista Singh

FaristaS · January 29, 2019, 4:58am

Hi mbacovsky,
Surely we can check by adding SHELL=/bin/bash in the script.
It will take time to share the foreman-maintain logs here, I am sure it will reveal much details or the cause of the problem.
Also will be sharing the cron logs here, will be helpful in troubleshooting.

Thanks
Farista Singh

FaristaS · January 29, 2019, 5:08am

Hi John,
I dont see the update version of rubygem-foreman_maintain.noarch in “foreman” repository.
Do i need to install separately or by downloading manually ?

John_Mitsch · January 29, 2019, 3:40pm

yum install -y https://yum.theforeman.org/latest/el7/x86_64/rubygem-foreman_maintain-0.3.0-1.el7.noarch.rpm should do install it fine on the system. The new version maintains compatibility with older katello versions, I gave it a quick test on a 3.8 box and didn’t see any issues

FaristaS · January 30, 2019, 5:03am

Hi mbacovsky,

I am attaching the foreman-maintain and cron logs for last night backup.
I have not updated foreman_maintain version to 0.3.0-1.el7.

Let me know if anything needs to be modified/installed

Thanks
Farista Singh

FaristaS · January 30, 2019, 9:24am

Attaching logs.
foreman-maintain.log.log (92.5 KB)
cron-script.log (1.1 MB)

Thanks
Farista Singh

FaristaS · February 1, 2019, 12:13pm

Hi There,

Please update with your analysis.

Regards
Farista Singh

mbacovsky · February 1, 2019, 11:43pm

Hi @FaristaS,
I checked the logs and was not able to find anything suspicious there. The only way I can think off how this could happen is the .snar files were not copied successfully from the previous backup.

Internally the procedure is as follow:

create new backup dir
copy there the .snar files from the referenced dir
…
do tar of the data with --listed-incremental dataname.snar

If the .snar file is missing the tar will silently create full backup and writes new .snar file.

In the logs we don’t have information about what .snar files are copied. Could you modify the tool to see what is going on there?

If so, please make a backup copy of the file being changed to be able to revert it back later.
The file can be found with rpm -ql rubygem-foreman_maintain|grep prepare_directory and should be something like /usr/share/gems/gems/foreman_maintain-0.2.4/definitions/procedures/backup/prepare_directory.rb on your system.
We need to add logging information before the copying happens. After the change the code should look like follows around line 25 (the four logger.debug lines were added):

      if @incremental_dir
        logger.debug('cp: ' + Dir.glob(File.join(@incremental_dir, '.*.snar')).to_s)
        logger.debug('backup_dir: ' + @backup_dir.to_s)
        logger.debug('tar: ' + `which tar`.to_s)
        logger.debug('ENV:' + ENV.inspect)
        FileUtils.cp(Dir.glob(File.join(@incremental_dir, '.*.snar')), @backup_dir)
      end

In the log the data could look like:

D, [2019-02-01 23:27:50+0000 #27385] DEBUG -- : cp: ["/tmp/satellite-backup-2019-02-01-22-15-38/.config.snar", "/tmp/satellite-backup-2019-02-01-22-15-38/.pulp.snar", "/tmp/satellite-backup-2019-02-01-22-15-38/.mongo.snar", "/tmp/satellite-backup-2019-02-01-22-15-38/.postgres.snar"]
D, [2019-02-01 23:27:50+0000 #27385] DEBUG -- : backup_dir: /tmp/satellite-backup-2019-02-01-23-27-40
D, [2019-02-01 23:27:50+0000 #27385] DEBUG -- : tar: /bin/tar

D, [2019-02-01 23:27:50+0000 #27385] DEBUG -- : ENV:{....}

If we can then compare the data in the logs run from the cron and the script we may reveal what is the cause of the problem.

Regards,
Martin

FaristaS · February 4, 2019, 9:15am

Hi Martin,

I have updated the /usr/share/gems/gems/foreman_maintain-0.2.4/definitions/procedures/backup/prepare_directory.rb file after taking backup of it.
Let’s wait to run it once and then I will share you the foreman-maintain log.

Thanks
Farista Singh

FaristaS · February 5, 2019, 5:13am

I have got the debug output file of foreman-maintain log.
Attaching it here,
Please analyse and revert.

Thanks
Farista Singh
foreman-maintain-debug.log (92.8 KB)