Katello 4.5, Foreman 3.3. Just upgraded from previous version.
The foreman backup now gets STUCK “re-enabling sync plans” at the end.
To clear this I have to shut everything down and reboot, and then re-enable the sync plan.
I had a similar issue in 4.3, fixed in 4.4. Possible regession?
As an aside using ''whitelist=… foor the sync plan choices does let me work around this, but I am hoping can still take a look at it’s clearly an issue. Thanks guys!
Ping. anyone?
If anyone running katello 4.5 can try doing a backup I’d really appreciate it (to see if this is actually a problem or SOMEHOW jusst me?)
Sample backup command:
foreman-maintain backup offline --assumeyes
Still zero response to my question (rats!).
Just adding a ping here to see if anyone has tried running a backup on or after Katello 4.5? Hoping this issue can be reproduced?
This is where is gets stuck.
Backup Foreman DB offline:
Already done [OK]
--------------------------------------------------------------------------------
Backup Pulpcore DB offline:
Already done [OK]
--------------------------------------------------------------------------------
Start applicable services:
Starting the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
/ All services started [OK]
--------------------------------------------------------------------------------
re-enable sync plans:
| re-enabling sync plans <== HERE
anyone? REALLY would like someone besides me trying a backup to see if this is reproducable…
It appears that I am encountering this issue in our QA testing environment.
Any feedback on how to further debug and resolve this issue?
I have gotten ZERO response to this issue which I’ve had for months now.
It seems that something with the restart of services isn’t executed as it should be.
I killed the foreman-maintain process after it reached the re-enable sync and was spinning it’s wheels there.
I then tried to run
foreman-maintain advanced procedure run sync-plans-enable
with the results
Running preparation steps required to run the next scenarios
================================================================================
Setup hammer:
Configuring Hammer CLI...
Hammer admin password:
[FAIL]
Hammer configuration failed: Incorrect credential for adminforeman user.
--------------------------------------------------------------------------------
Scenario [preparation steps required to run the next scenarios] failed.
The following steps ended up in failing state:
[hammer-setup]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="hammer-setup"
If I rebooted the server instead, everything is work as it should after the server starts the Foreman services.
foreman-maintain advanced procedure run sync-plans-enable
Running ForemanMaintain::Scenario
================================================================================
re-enable sync plans:
- Total 7 sync plans are now enabled. [OK]
--------------------------------------------------------------------------------
I’m going to continue playing with this and hopefully support takes some interest in this.
I get the impression that the system is unable to access it’s configuration for hammer until the system is rebooted and then services are normal and hammer works as it should.
I run in the same issue:
D, [2023-01-30 11:39:19+0100 #4690] DEBUG -- : Running command PGPASSWORD='[FILTERED]' psql -h localhost -p 5432 -U foreman -d foreman with stdin "COPY ( select sp.id as id from katello_sync_plans sp inner join foreman_tasks_recurring_logics rl on sp.foreman_tasks_recurring_logic_id = rl.id\n where rl.state='active' AND sp.id IN ('1','2','3','4')\n) TO STDOUT WITH CSV HEADER"
D, [2023-01-30 11:39:19+0100 #4690] DEBUG -- : output of the command:
id
I, [2023-01-30 11:39:23+0100 #4690] INFO -- : Hammer setup is not valid. Fixing configuration.
D, [2023-01-30 11:39:23+0100 #4690] DEBUG -- : Running command hostname -f with stdin nil
D, [2023-01-30 11:39:23+0100 #4690] DEBUG -- : output of the command:
or.dev.atix
I, [2023-01-30 11:39:26+0100 #4690] INFO -- : Invalid admin password was found in hammer configs. Looking into installer answers
hammer is not able to reach the API. Additonally, no access to the UI via browser possible (connection refused).
Service restart did not help. As @frostygresh said, reboot of the whole system worked.
This fixes the issue:
Fix confirmed; I see in Katello 4.6 (I am at 4.6.2) the backup now works and does not fail on re-enabling the sync plan. Yay!
Starting the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, p-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@works-queue-1, foreman-proxy
\ All services started [OK]
--------------------------------------------------------------------------------
re-enable sync plans:
- Total 1 sync plans are now enabled. [OK]
--------------------------------------------------------------------------------
Remove maintenance mode table/chain from nftables/iptables: [OK]
--------------------------------------------------------------------------------
Compress backup data to save space:
/ Compressing backup of Postgres DB [OK]
--------------------------------------------------------------------------------
Done with backup: 2023-02-07 14:15:46 -0600
**** BACKUP Complete, contents can be found in: /backups/katello-backup-2023-02-07-13-57-55 ****