Katello 4.2.1 foreman-maintain backup missing critical files

Hi and Happy New Year,

I am hoping someone out there knows what may be the problem here.

Problem:

Our new Katello 4.2.1 server is not able to take valid backups with foreman-maintain.

The offline backups don’t contain the mongo_data.tar.gz or the pulp tar ball.

When I go to restore I get the following error:

root@katello04-> foreman-maintain restore /var/katello_backups/katello04/katello-backup-2022-01-07-12-59-18/
Running Restore backup

Check if command is run as root user: [OK]

Validate backup has appropriate files: [FAIL]

The given directory does not contain the required files or has too many files

All backup directories contain: config_files.tar.gz
An online or remote database backup directory contains: mongo_dump, candlepin.dump, foreman.dump, pulpcore.dump
An offline backup directory contains: mongo_data.tar.gz, pgsql_data.tar.gz
A logical backup directory contains: mongo_dump, candlepin.dump, foreman.dump, pulpcore.dump, mongo_data.tar.gz, pgsql_data.tar.gz
Including pulp_data.tar is optional and will restore pulp data to the filesystem if included.

Only the following files were found: pgsql_data.tar.gz, config_files.tar.gz, metadata.yml

Expected outcome:

The foremain-maintain offline backups should produce valid backups with all the requisite files needed to do a restore

Our directory just has the following files in there:

root@katello03-> ll /var/katello_backups/katello04/katello-backup-2022-01-07-12-59-18/
total 445988
-rw-r–r–. 1 root root 2371127 Jan 7 13:00 config_files.tar.gz
-rw-r–r–. 1 root root 39136 Jan 7 12:59 metadata.yml
-rw-r–r–. 1 root root 454275885 Jan 7 13:00 pgsql_data.tar.gz
[/var/katello_backups/katello04/katello-backup-2022-01-07-12-59-18]
root@katello03->


An older version of Katello (3.18) has all the files needed and we can restore fine:

root@katello01-> ll
total 122460680
-rw-r–r–. 1 root root 8427463 Nov 5 17:21 config_files.tar.gz
-rw-r–r–. 1 root root 46021 Nov 5 17:20 metadata.yml
-rw-r–r–. 1 root root 95472160 Nov 5 17:26 mongo_data.tar.gz
-rw-r–r–. 1 root root 1635086649 Nov 5 17:26 pgsql_data.tar.gz
-rw-r–r–. 1 root root 123660687360 Nov 5 17:26 pulp_data.tar
[/var/katello_backups/katello01/katello-backup-2021-11-05-17-20-15]
Foreman and Proxy versions:

Foreman 3.0.1 and Katello 4.2.1

Foreman and Proxy plugin versions:

Distribution and version:

root@katello03-> cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

I did some looking around and it appears that mongoDB isn’t part of Foreman 3.x and Katello 4.x anymore as I don’t see it running in the process list:

root@katello03-> ps auxwww | grep -i mongo
root 19288 0.0 0.0 112824 2320 pts/0 S+ 18:18 0:00 grep --color=auto -i mongo
[/var/katello_backups/katello03/katello-backup-2022-01-11-18-06-08]
root@katello03->

also doesn’t look like any mongo packages are installed:

root@katello03-> rpm -qa | grep -i mongo
[/var/katello_backups/katello03/katello-backup-2022-01-11-18-06-08]
root@katello03->

So one thing I think is that the foreman-maintain backup script shouldn’t be requiring the mongo tar ball but the other thing is it should still dump the pulp tar ball

What version of foreman-maintain do you have? 0.9 and newer should be able to handle Pulp3 without MongoDB just fine.

Actually, I was wrong, 0.8.11 was the version that supports it.

hi @evgeni and @gluciano,

Thank you so much for replying.

So I think both gluciano and myself are having the same issue. I am running Foreman 3.0 and Katello 4.2

The rpm that was installed from a fresh install of katello was:

root@katello03-> rpm -qa | grep -i maintain
rubygem-foreman_maintain-0.8.10-1.el7.noarch
[~]
root@katello03->

Would you please be able to speak on how best to upgrade this RPM? I have cloned the git repo from here:

root@katello03-> git clone GitHub - theforeman/foreman_maintain: The Foreman/Satellite maintenance tool

I think that cloned the latest tagged version of 0.8.22 so another question is it safe to use the latest version of foreman_maintain on an older version of foreman / katello? Meaning, is it smart enough to do the right things even though we aren’t on Foreman 3.1 and Katello 4.3?

Cheers,

Eledor

root@uscdc01plsat001:~# rpm -qa | grep -i maintain
rubygem-foreman_maintain-0.8.6-1.el7.noarch

This is what version I am on. Yum updates do not provide any updates for this.

Yep, same here

I looked at the Foreman repos and for the version of Foreman we are on the RPM is here:

https://yum.theforeman.org/releases/3.0/el7/x86_64/

rubygem-foreman_maintain-0.8.10-1.el7.noarch.rpm
rubygem-foreman_maintain-doc-0.8.10-1.el7.noarch.rpm


To get to a version of foreman-maintain that supports our version of Foreman / Katello we would have to use this repo for 3.1:

https://yum.theforeman.org/releases/3.1/el7/x86_64/

rubygem-foreman_maintain-0.9.2-1.el7.noarch.rpm
rubygem-foreman_maintain-doc-0.9.2-1.el7.noarch.rpm

But I am not entirely sure how safe that would be to do. Meaning, remove the old version of the package with yum and then install the new package with yum. It may work but I am concerned about the integrity of the backups.

Hopefully Evgeni can shine some light on this.

Thx,

Eledor

Hmm… when trying to remove the foreman_maintain package it looks like it also wants to rip out the katello main package as well:

root@katello03-> yum remove rubygem-foreman_maintain-0.8.10-1.el7.noarch
Loaded plugins: fastestmirror, rhnplugin, tsflags, versionlock
This system is not registered with RHN Classic or Red Hat Satellite.
You can use rhn_register to register.
Red Hat Satellite or RHN Classic support will be disabled.
Resolving Dependencies
→ Running transaction check
—> Package rubygem-foreman_maintain.noarch 1:0.8.10-1.el7 will be erased
→ Processing Dependency: rubygem-foreman_maintain >= 0.2.2 for package: katello-4.2.1-1.el7.noarch
→ Running transaction check
—> Package katello.noarch 0:4.2.1-1.el7 will be erased
→ Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================================================
Package Arch Version Repository Size

Removing:
rubygem-foreman_maintain noarch 1:0.8.10-1.el7 @foreman 529 k
Removing for dependencies:
katello noarch 4.2.1-1.el7 @katello 232

Transaction Summary

Remove 1 Package (+1 Dependent package)

Installed size: 530 k
Is this ok [y/N]: n

Hopefully there is a workaround for this… we have stood up parallel installs of Katello to fix the smart proxy sync issues on 3.18 and were hoping 4.2 would be the answer

I’ve pushed out 0.8.21 to the Foreman 3.0 repos, please try that :slight_smile:

As for integrity, you need to create the backup with 0.8.11+, and then it should restore fine on (currently) any later version (I say currently, as I don’t think we give any guarantee that the contents won’t change, but I think we didn’t do any changes that should affect that in the last months).

thank you very much @evgeni . that new RPM allowed us to back things up and do a restore fine on the same host. to make doubly sure on things i am going to try a restore on a separate VM but all this looks positive.

cheers,

Eledor

1 Like

moving to a completely separate VM also worked.

thanks @evgeni

hope that helped you too @gluciano

nice, glad it worked out for you.

I upgraded to 3.1, the backup worked but during a restore I get the following error, not sure what this is about now:

Extract any existing tar files in backup:
\ Extracting pgsql data [OK]

Migrate pulpcore db:
| starting postgresql
Warning: postgresql.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.

  • Migrating pulpcore database [FAIL]
    Failed executing sudo PULP_SETTINGS=/etc/pulp/settings.py DJANGO_SETTINGS_MODULE=pulpcore.app.settings python3-django-admin migrate --noinput, exit status 1:
    Traceback (most recent call last):
    File “/usr/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run_from_argv
    self.execute(*args, **cmd_options)
    File “/usr/lib/python3.6/site-packages/django/core/management/base.py”, line 361, in execute
    self.check()
    File “/usr/lib/python3.6/site-packages/django/core/management/base.py”, line 390, in check
    include_deployment_checks=include_deployment_checks,
    File “/usr/lib/python3.6/site-packages/django/core/management/commands/migrate.py”, line 64, in _run_checks
    issues = run_checks(tags=[Tags.database])
    File “/usr/lib/python3.6/site-packages/django/core/checks/registry.py”, line 72, in run_checks
    new_errors = check(app_configs=app_configs)
    File “/usr/lib/python3.6/site-packages/django/core/checks/database.py”, line 9, in check_database_backends
    for conn in connections.all():
    File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 216, in all
    return [self[alias] for alias in self]
    File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 213, in iter
    return iter(self.databases)
    File “/usr/lib/python3.6/site-packages/django/utils/functional.py”, line 80, in get
    res = instance.dict[self.name] = self.func(instance)
    File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 147, in databases
    self._databases = settings.DATABASES
    File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 79, in getattr
    self._setup(name)
    File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 66, in _setup
    self._wrapped = Settings(settings_module)
    File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 157, in init
    mod = importlib.import_module(self.SETTINGS_MODULE)
    File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
    File “”, line 994, in _gcd_import
    File “”, line 971, in _find_and_load
    File “”, line 941, in _find_and_load_unlocked
    File “”, line 219, in _call_with_frames_removed
    File “”, line 994, in _gcd_import
    File “”, line 971, in _find_and_load
    File “”, line 941, in _find_and_load_unlocked
    File “”, line 219, in _call_with_frames_removed
    File “”, line 994, in _gcd_import
    File “”, line 971, in _find_and_load
    File “”, line 953, in _find_and_load_unlocked
    ModuleNotFoundError: No module named ‘pulpcore’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/bin/python3-django-admin”, line 11, in
load_entry_point(‘Django==2.2.24’, ‘console_scripts’, ‘django-admin’)()
File “/usr/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line
utility.execute()
File “/usr/lib/python3.6/site-packages/django/core/management/init.py”, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/usr/lib/python3.6/site-packages/django/core/management/base.py”, line 336, in run_from_argv
connections.close_all()
File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 219, in close_all
for alias in self:
File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 213, in iter
return iter(self.databases)
File “/usr/lib/python3.6/site-packages/django/utils/functional.py”, line 80, in get
res = instance.dict[self.name] = self.func(instance)
File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 147, in databases
self._databases = settings.DATABASES
File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 79, in getattr
self._setup(name)
File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 66, in _setup
self._wrapped = Settings(settings_module)
File “/usr/lib/python3.6/site-packages/django/conf/init.py”, line 157, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named ‘pulpcore’

Scenario [Restore backup] failed.

The following steps ended up in failing state:

[pulpcore-migrate]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“pulpcore-migrate”

Running Rescue Restore backup

is that a freshly installed 3.1 where you’re trying to restore to?

Yes, it was just upgraded from 2.5 to 3.1, I did a backup of 3.1 and then tried the restore. I just did it again with a new backup and tried the restore on the new backup. Same error message as above.