Problem:
Foreman and Katello version upgraded from Foreman 3.10 and Katello 4.12 to Foreman 3.11 and Katello 4.13
When Upgrading,
dnf -y module switch-to postgresql:13
command moves postgresql service to failed state.
After updating locale from en_US.UTF-8 to C.UTF-8 using below command the foreman-installer completed without issues,
localectl set-locale C.UTF-8
After Foreman upgrade all the services are running good and able to use the UI without any problem.
But when trying to run Foreman backup online using the command,
foreman-maintain backup online /var
Backup runs at the begenning and it is struck on,
Backup Pulp data: - Collecting Pulp data line
There is an No disk space issue raised on vm eventhough it had around 800GB space left. Expected outcome:
Backup command runs successfully. Foreman and Proxy versions:
Foreman 3.11 and Katello 4.13 Distribution and version:
OS: Rocky Linux release 8.10 (Green Obsidian) Other relevant data:
VM Spec:
CPU: 8
RAM: 40GB
Disk: 2TB
Size of products and content views: 550GB
Were you using online backups before successfully?
Does an offline backup produce the same results?
Can you post the whole output of the backup process please?
Thats rubygem-foreman_maintain 1.6.9 you are running, right?
Our requirement is to take backup without any downtime in production server,so online backup is being followed,will check on offline backup and update you.
Online backup:
[root@foreman ~]# foreman-maintain backup online /var
Starting backup: 2024-09-27 11:26:09 +0530
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up:
/ Checking connection to the Foreman DB [OK]
--------------------------------------------------------------------------------
Make sure Candlepin DB is up:
- Checking connection to the Candlepin DB [OK]
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up:
\ Checking connection to the Pulpcore DB [OK]
--------------------------------------------------------------------------------
Running Backup
================================================================================
Check if the incremental backup has the right type: [OK]
--------------------------------------------------------------------------------
Check for running tasks: [OK]
--------------------------------------------------------------------------------
Check for running pulpcore tasks: [OK]
--------------------------------------------------------------------------------
Prepare backup Directory:
Creating backup folder /var/katello-backup-2024-09-27-11-26-09 [OK]
--------------------------------------------------------------------------------
Generate metadata:
| Saving metadata to metadata.yml [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):
pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpc ore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-w orker@7.service, pulpcore-worker@8.service, dynflow-sidekiq@worker-1, dynflow-sidekiq@ worker-hosts-queue-1
/ All services stopped [OK]
--------------------------------------------------------------------------------
Backup config files:
/ Collecting config files to backup [OK]
--------------------------------------------------------------------------------
Backup Pulp data:
\ Collecting Pulp data
[root@foreman ~]# foreman-maintain backup offline /var
Starting backup: 2024-09-24 20:40:52 +0530
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up:
/ Checking connection to the Foreman DB [OK]
--------------------------------------------------------------------------------
Make sure Candlepin DB is up:
- Checking connection to the Candlepin DB [OK]
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up:
\ Checking connection to the Pulpcore DB [OK]
--------------------------------------------------------------------------------
Running Backup
================================================================================
Check if the incremental backup has the right type: [OK]
--------------------------------------------------------------------------------
Check for running tasks: [OK]
--------------------------------------------------------------------------------
Check for running pulpcore tasks: [OK]
--------------------------------------------------------------------------------
Confirm turning off services is allowed:
WARNING: This script will stop your services.
Do you want to proceed?, [y(yes), q(quit)] y
[OK]
--------------------------------------------------------------------------------
Prepare backup Directory:
Creating backup folder /var/katello-backup-2024-09-24-20-40-52 [OK]
--------------------------------------------------------------------------------
Generate metadata:
/ Saving metadata to metadata.yml [OK]
--------------------------------------------------------------------------------
Detect features available in the local proxy: [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-api.socket, pulpcore-content.socket, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
- All services stopped [OK]
--------------------------------------------------------------------------------
Backup config files:
/ Collecting config files to backup [OK]
--------------------------------------------------------------------------------
Backup Pulp data:
/ Collecting Pulp data
Backup is stopped here.
Disk details before the vm got struck:
“good”, well, at least in the sense of “this is not caused by our recent changes to online backups”.
I don’t think so, no.
When the backup is stuck at the “Collecting Pulp data”, I’d expect there is a process running tar on /var/lib/pulp, can you check that such a process is running and actually reading and writing data?
I would assume that process from the 22 is hanging.
Can you try killing it and see if a backup would now proceed?
If not I’d be interested what the (new) tar process is doing.
Date and time was not synced,I took the command output by today with todays date as 22-09-2024 on the VM.Now I have synced the time so it is showing correctly.
One general doubt regarding online backup,
Pulp services are stopped during the backup,so new repos creation and sync are not available till the backup completion in latest foreman version?
Thats why you said online backups can be used for production in latest version of foreman and katello?
