Unsuccessful upgrade from F2.1+K3.16 to F2.2+K3.17, every time

Problem:

Every time I attempt to upgrade Foreman from 2.1 to 2.2 (and Katello from 3.16 to 3.17), it fails and results an an unusable stack.

Expected outcome:

A functional Foreman & Katello stack.

Foreman and Proxy versions:

Before:

$ rpm -q foreman katello
foreman-2.1.4-1.el7.noarch
katello-3.16.2-1.el7.noarch

After:

$ rpm -q foreman katello
foreman-2.2.3-1.el7.noarch
katello-3.17.3-1.el7.noarch

Foreman and Proxy plugin versions:

The standard set of plugins that come with the base install. I haven’t installed any plugins myself. Any dependencies came along for the ride during the upgrade.

However, I did notice that the remote execution plugin got installed as part of the upgrade (though I didn’t specify it). Is it a new dependency? Any time I’ve tried installing it in the past, I was unsuccessful, as it made my Foreman+Katello install unusable and I had to roll back.

Distribution and version:

CentOS Linux release 7.9.2009 (Core)

Other relevant data:

These are the errors summarized at the end of /var/log/foreman-installer/katello.log:

[ERROR 2021-04-27T16:52:59 main] Errors encountered during run:
[ERROR 2021-04-27T16:52:59 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
[ERROR 2021-04-27T16:52:59 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
[ERROR 2021-04-27T16:52:59 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
[ERROR 2021-04-27T16:52:59 main]  Command exceeded timeout
[ERROR 2021-04-27T16:52:59 main]  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: Command exceeded timeout
[ERROR 2021-04-27T16:52:59 main]  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Failed to call refresh: Command exceeded timeout
[ERROR 2021-04-27T16:52:59 main]  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Command exceeded timeout

…and these errors get repeated indefinitely in a loop in /var/log/foreman/production.log:

2021-04-28T18:17:04 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2021-04-28T18:17:05 [W|app|] Scoped order is ignored, it's forced to be batch order.
2021-04-28T18:17:05 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2021-04-28T18:17:05 [W|app|] Could not create role 'Remote Execution Manager': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_audit_logs", "view_hosts", "view_smart_proxies", "view_job_templates", "create_job_templates", "edit_job_templates", "edit_remote_execution_features", "destroy_job_templates", "lock_job_templates", "create_job_invocations", "view_job_invocations", "create_template_invocations", "cancel_job_invocations", "filter_autocompletion_for_template_invocation", :view_job_templates, :view_job_invocations, :create_job_invocations, :create_template_invocations, :view_hosts, :view_smart_proxies, :cancel_job_invocations, :destroy_job_templates, :edit_job_templates, :create_job_templates, :lock_job_templates, :view_audit_logs, :filter_autocompletion_for_template_invocation, :edit_remote_execution_features]
2021-04-28T18:17:05 [E|app|] Cannot continue because some permissions were not found, please run rake db:seed and retry
2021-04-28T18:17:05 [I|app|] Rails cache backend: File

Does anyone have any insights that would help me figure-out what’s broken about my installation?

Did you try what the error message suggests?

Sorry, I forgot to mention that. I did run rake db:seed, but that didn’t make any difference. It just resulted in a different error, which I couldn’t get past. I’ll run through it again and post the results here, later.

Hi @pdelong42

Can you run the rake db:seed with the --trace option and post the output?

Did the db:migration work ok?

In a nutshell I get the “User with login admin already exists, not seeding as admin.” error from trying a db:seed. I also ran the db:migrate, and it seemed to run without any issues (since the first attempt produced no output, I ran it again with --trace and checked the exit code, FWIW). See the detailed output below:

$ cat try.sh
#!/bin/sh

# Usage: script -c 'sudo sh try.sh' $(date +%F_%T).log

set -xeuo pipefail

foreman-rake db:seed --trace
foreman-rake db:migrate

$ script -c 'sudo sh try.sh' $(date +%F_%T).log
Script started, file is 2021-04-29_11:40:35.log
+ foreman-rake db:seed --trace
** Invoke db:seed (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Execute db:seed
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke db:load_config 
** Execute db:abort_if_pending_migrations
** Invoke dynflow:abort_if_pending_migrations (first_time)
** Invoke environment 
** Execute dynflow:abort_if_pending_migrations
User with login admin already exists, not seeding as admin.
+ foreman-rake db:migrate
Script done, file is 2021-04-29_11:40:35.log
$ sudo foreman-rake db:migrate --trace
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke plugin:refresh_migrations (first_time)
** Invoke environment 
** Execute plugin:refresh_migrations
** Execute db:migrate
** Invoke db:_dump (first_time)
** Execute db:_dump
** Invoke dynflow:migrate (first_time)
** Invoke environment
** Execute dynflow:migrate
$ echo $?
0

I did also find this related issue (which left me none the wiser):

…though isn’t entirely clear to me whether that fellow actually ever got over the hurdle he’s described.

What other helpful diagnostic data can I provide? I’m stopping short of just uploading a tarball of my /var/log directory, seeing as this is a public forum.

Also, would there be any merit in trying to upgrade directly to Foreman v2.3 and Katello v3.18 (on the off chance that the bugs I’m tripping over have been fixed in that version of the installer)? The phrasing used in the upgrade documentation seems a bit ambiguous on that count:

Katello supports upgrades from the previous two versions only. Upgrades should be performed sequentially without skipping versions in between.

Is it just me, or do those two sentences contradict each-other? (cf. Foreman :: Plugin Manuals)

Hi @pdelong42

Can you generate a foreman-debug and email it to me at chrobert @ redhat.com? I will take a look today.

Thanks @cintrix84 , I’ll be sending it to you shortly (in the next five minutes). I’ve included the output of foreman-debug runs for both before and after the upgrade (in case a wonky condition existed prior to upgrading).

1 Like

Hi @cintrix84 ,

Any luck? I hope the email found you well (I didn’t get a bounce, anyway).

I do appreciate you having a look, seeing as I’m on the free product.