Katello 3.16 upgrade stops on 'Executing: foreman-rake upgrade:run'

Support
, , ,
1

I’m doing a march of upgrades to get from Foreman 1.23.2/Katello 3.13.4 up to current release.

I’ve made it to 2.1.4/3.16.2 and I’m facing a dead stop with foreman-install. It just hangs with no obvious errors. I’ve tried twice and let sit for 2-3 hours. The UI is up and appears functional.

The tail of foreman-installer/katello.log shows:

[DEBUG 2021-10-05T11:52:17 main]  /File[/opt/puppetlabs/puppet/cache/facts.d]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache]
[DEBUG 2021-10-05T11:52:17 main]  /File[/opt/puppetlabs/puppet/cache/locales]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache]
[DEBUG 2021-10-05T11:52:17 main]  Finishing transaction 75966100
[DEBUG 2021-10-05T11:52:17 main]  Received report to process from foreman01-ops.am-usw02.ttops.net
[ INFO 2021-10-05T11:52:18 main] Puppet has finished, bye!
[ INFO 2021-10-05T11:52:18 main] Executing hooks in group post
[DEBUG 2021-10-05T11:52:18 main] Executing: foreman-rake upgrade:run

Any guidance appreciated!

Best,
Matt

2

Is there any activity in /var/log/foreman/production.log while foreman-rake upgrade:run is executing? If you monitor top while it’s running, does the system appear idle?

3

And of course as soon as I go back to look and provide details, it’s finished it’s run! Does have an error I need to chase down.

Previous upgrades had giving long pauses at the forewarned Upgrade Step 1/3: katello:correct_repositories and I did just wait. As you can see from timestamps, this time it paused before that warning and didn’t update until 2 hours later. An aside, a previous update also paused for almost exactly 2 hours. What’s up with that?

The command results show:

foreman-rake upgrade:run finished successfully!
  Success!
  * Foreman is running at https://foreman01-ops.am-usw02.ttops.net

The logs shows:

[ INFO 2021-10-05T11:52:18 main] Executing hooks in group post
[DEBUG 2021-10-05T11:52:18 main] Executing: foreman-rake upgrade:run
[DEBUG 2021-10-05T13:52:38 main] =============================================
[DEBUG 2021-10-05T13:52:38 main] Upgrade Step 1/3: katello:correct_repositories. This may take a long while.
[DEBUG 2021-10-05T13:52:38 main] Processing Repository 1/215: CentOS7Base (1)
[DEBUG 2021-10-05T13:52:38 main] Processing Repository 2/215: CentOS7EPEL (2)
[snip]
[ INFO 2021-10-05T13:52:38 main] All hooks in group post finished
[DEBUG 2021-10-05T13:52:38 main] Exit with status code: 2 (signal was 2)
[ERROR 2021-10-05T13:52:38 main] Errors encountered during run:
[ERROR 2021-10-05T13:52:38 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
[DEBUG 2021-10-05T13:52:38 main] Cleaning /tmp/kafo_installation20211005-4159-14dqir7
[DEBUG 2021-10-05T13:52:38 main] Cleaning /tmp/kafo_installation20211005-4159-qzjjb8
[DEBUG 2021-10-05T13:52:38 main] Cleaning /tmp/default_values.yaml
[ INFO 2021-10-05T13:52:38 main] Installer finished in 7285.896715591 seconds

To answers your other question, the host is very low load during this time.

4

Ok, glad it completed! Yeah, that particular task (correct repositories) can take some time. It may go faster if you remove any unused repos any old Content View Versions from your installation.

Regarding the foreman-maintain packages is-locked error, I think that’s OK to ignore as long as the installer itself completed OK (which looks like it did). I think foreman-maintain was fixed in a later version. If you’ve done a sanity check of your environment (hammer ping is a good one to verify too) then I’d say it’s safe to continue to the next version.

1 Like