I have two katello installs (same version 4.3.1). The other night I did some yum updates followed by a foreman-installer; no issues .
But now I have ended up with both installs STUCK in maintenance mode. Help?
On node A, I was doing a katello backup. It got stuck trying to enabe the sync plan, so I had to kill it. But it’s in maintnance mode, and I cannot get out. I am getting asked for a password.
[root@katello ~]# foreman-maintain maintenance-mode stop
Running preparation steps required to run the next scenarios
Setup hammer:
Configuring Hammer CLI…
Hammer admin password:
[FAIL]
Hammer configuration failed: Incorrect credential for admin user.
Scenario [preparation steps required to run the next scenarios] failed.
The following steps ended up in failing state:
[hammer-setup]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“hammer-setup”
So, I tried entering my admin password. Does not work. I verified it’s listed correctly in ~/.hammer/cli.modules.d/foreman.yml
no dice. Also: the “–whitelist=…” is an invalid syntax for the foreman-maintain command.
Out of curiosity, I went to my other katello node, still running. I simply did “foremain-maintain maintenance-mode start”. This worked. No password needed. BUT: When I tried Stopping maintenance mode… SAME THING. I am being asked for a password. My admin password does not work.
What the heck ? How can I get out of maintenance mode?
Interestingly enough, “foreman-maintain maintenance-mode status”, just like “start”… does not run the hammer-cli thing that requires the password.
# foreman-maintain maintenance-mode status
Running Status of maintenance-mode
================================================================================
Check of maintenance-mode consistency:
/ Running status of maintenance-mode
Status of maintenance-mode: On
- Nftables table: present
- sync plans: disabled
- cron jobs: not managed [OK]
--------------------------------------------------------------------------------
But the “stop” subcommand is failing and I am stuck. Yikes. Hope someone has an idea 
In the example above and allso here, I am manually entering the correct password btw.
# foreman-maintain maintenance-mode stop
Running preparation steps required to run the next scenarios
================================================================================
Setup hammer:
Configuring Hammer CLI...
Hammer admin password:
[FAIL]
Hammer configuration failed: Incorrect credential for admin user.
--------------------------------------------------------------------------------
Scenario [preparation steps required to run the next scenarios] failed.
The following steps ended up in failing state:
[hammer-setup]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="hammer-setup"
To quote Chris O’Dowd (“Roy” from “The IT Crowd”): “Have you tried turning it off and on again?”
A server reboot has cleared the condition. GO FIGURE.
Attempting a backup again now.
“Nothing to see here. Move along…”
UPDATE:
OK, I jumped the gun a little on my conclusion. There definitely is a problem with maintenance mode.
Putting foreman INTO mainternance mode: no problem.
Stopping mainternance-mode: I get the same behavior, I eventually get asked for a password (and correct pw does not work).
I uncovered this situation by attempting an offline backup. It gets stuck trying to use hammer to re-enable the sync plan.
Further information:
For me, the only way to fix the hosed-up state is to first reboot, then run maintenance-mode status. Foreman recognizes it’s out of whack, but then-- it continues and things are finally back up and running… the re-enbling of the sync plan works HERE. But not earlier. Whoa.
# foreman-maintain maintenance-mode status
Running Status of maintenance-mode
================================================================================
Check of maintenance-mode consistency:
/ Running status of maintenance-mode
Status of maintenance-mode: Off
- Nftables table: absent
- sync plans: disabled
- cron jobs: not managed [FAIL]
You can follow remediation procedure(s) to fix maintenance-mode state
--------------------------------------------------------------------------------
Continue with step [re-enable sync plans]?, [y(yes), n(no), q(quit)] y
re-enable sync plans:
\ Total 1 sync plans are now enabled. [OK]
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check of maintenance-mode consistency:
| Running status of maintenance-mode
Status of maintenance-mode: Off
- Nftables table: absent
- sync plans: enabled
- cron jobs: not managed [OK]
--------------------------------------------------------------------------------
Just completed upgrading to foreman 3.2 / Katello 4.4.
This appears to no longer be an issue. Might affect some 3.1/4.3 users, FYI.
2 Likes
