Problem:
Running foreman-installer after updating the packages to katello 3.16 or 3.17 from 3.15 using their respective foreman 2.1 or 2.2 versions fail where the installer is doing the foreman-maintain packages locked check.
Release notes or upgrade doco do not mention any action a customer can take to upgrade from 3.15, and doco specifically say upgrades of up to 2 versions only is supported.
So how would one upgrade to say 3.17 from 3.15, if the fix was not backported to 3.16 or 3.17?
Expected outcome:
Running foreman-installer works without an error during upgrade to katello 3.16 or 3.17 from 3.15.
Foreman and Proxy versions:
2.0.3 Foreman and Proxy plugin versions:
katello 3.15
Foreman installer output says there is an issue with db migrate:
root@vemplnmgmt4001 /etc/yum.repos.d # foreman-installer
‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from ‘notrun’ to [‘0’] failed: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
Installing Done [100%] […]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/foreman-installer/katello.log
Tail of the installer log, indicates it is because packages is-locked returned a bad exit code, and that is why db migrate did not run:
root@vemplnmgmt4001 /etc/yum.repos.d # tail -12 /var/log/foreman-installer/katello.log |more
[ INFO 2021-07-20T22:38:14 main] All hooks in group post finished
[DEBUG 2021-07-20T22:38:14 main] Exit with status code: 6 (signal was 6)
[ERROR 2021-07-20T22:38:14 main] Errors encountered during run:
[ERROR 2021-07-20T22:38:14 main] foreman-maintain packages is-locked --assumeyes failed! Check the output fo
r error!
[ERROR 2021-07-20T22:38:14 main] ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
[ERROR 2021-07-20T22:38:14 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake
-db:migrate]/returns: change from ‘notrun’ to [‘0’] failed: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 i
nstead of one of [0]
[ERROR 2021-07-20T22:38:14 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake
-db:migrate]: Failed to call refresh: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
[ERROR 2021-07-20T22:38:14 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake
-db:migrate]: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
[DEBUG 2021-07-20T22:38:14 main] Cleaning /tmp/kafo_installation20210720-7987-8po692
[DEBUG 2021-07-20T22:38:14 main] Cleaning /tmp/kafo_installation20210720-7987-15z3nts
[DEBUG 2021-07-20T22:38:14 main] Cleaning /tmp/default_values.yaml
[ INFO 2021-07-20T22:38:14 main] Installer finished in 145.372339253 seconds
Running the package check manually returns a success return code, and also output confirms there is no issue. This is in line with the bug fix for the installer I referenced.
Running db migrate manually, is fine, no errors.
I can’t upload logs.
From what I ma seeing, the issue is exactly what was reported in the bug I referenced and which was fixed in foreman 2.3
Sorry, but this looks nothing like bug 31135. You are in a completely different place in your logs. The bug is about an issue in the very early stages of the installer.
Your problem is the failure of /usr/sbin/foreman-rake db:migrate
Are you sure you can execute this and don’t get any error?
Can you run
# /usr/sbin/foreman-rake db:migrate --trace
Are you sure you have the correct repositories enabled for the upgrade to 3.16? In particular the correct pulpcore? Check with “yum repolist -v”.
Yes db migrate runs fine and postgresql-evr package is installed.
As I said, the installer bails at the package locked check, which returns exit code 1, even when packages are not locked.
I found in log more details which implies and issue with ruby gems.
[DEBUG 2021-07-20T23:03:38 main] /usr/share/rubygems/rubygems/dependency.rb:296:in to_specs': Could not find 'foreman_maintain' (>= 0) among 226 total gem(s) (Gem::LoadError) [DEBUG 2021-07-20T23:03:38 main] from /usr/share/rubygems/rubygems/dependency.rb:307:in to_spec’
[DEBUG 2021-07-20T23:03:38 main] from /usr/share/rubygems/rubygems/core_ext/kernel_gem.rb:47:in gem' [DEBUG 2021-07-20T23:03:38 main] from /usr/bin/foreman-maintain:22:in ’
[ERROR 2021-07-20T23:03:38 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
Here is more info after more digging.
foremain-maintain is-locked will return 0 if the packages is locked and 1 if it is not locked.
–assumeyes, here has no relevance, and in my view is a bug.
From that I will assume then that the installer wants to lock the packages and it fails when it checks it. In my config file, locking packages is blank.
So then I tried this, to specifically tell the installer I want to lock the packages
root@vemplnmgmt4001 /var/log/foreman-installer # foreman-installer --lock-package-versions
Locking of package versions was requested but foreman-maintain version installed does not support it
Not true, it is supported according to the help, which is the exact command the installer also use to determine if locking is supported.
root@vemplnmgmt4001 /var/log/foreman-installer # foreman-maintain packages -h
Usage:
foreman-maintain packages [OPTIONS] SUBCOMMAND [ARG] …
Subcommands:
lock Prevent packages from automatic update
unlock Enable packages for automatic update
status Check if packages are protected against update
install Install packages in an unlocked session
update Update packages in an unlocked session
is-locked Check if update of packages is allowed
Options:
-h, --help print help
So then I tried rerunning the installer but telling it not to bother locking packages.
root@vemplnmgmt4001 /var/log/foreman-installer # foreman-installer --no-lock-package-versions
‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
And it bails for same reason, exit code for the lock check is a fail, which is 1 since packages is not locked.
They only way I managed to get Katello to version 3.17 is the following, given the fix I need for foreman-installer was done in foreman 2.3 (Katello 3.18).
I installed the yum repos for katello 3.17 (incl foreman 2.2) as per upgrade guide.
Upgraded the packages
Ran foreman-installer and it did all the things and stopped at the db:migrate because of the lock package bug as per above in my post.
I updated the foreman yum repo to 2.3, and upgraded only foreman-installer and foreman-installer-katello
Then reverted the yum repo back to 2.2, and reran foreman-installer, it then completed the db migration and all the post upgrade parts.
Only issue I bumped into was that I then bump into a issue where the foreman-proxy is seemingly missing the “registration” feature, and it is expecting it, but it is not installing/enabling it either.
Again the usual stuff like restarting services does not work.
Don’t have an answer for this part yet, but so far if I ignore that installer error, rest of the app suit seems to work, and all components seems to be upgraded to expected components.
I managed now to upgrade to katello 3.18, from 3.17, and with that upgrade, it installed/configured the missing registration feature for foreman-proxy.
So obviously this was introduced somewhere in 3.17 but then only really fixed in 3.18.
# foreman-maintain packages -h