Foreman 2.5 -> 3.0 upgrade fails on db:migrate "NameError: uninitialized constant Environment"

Problem:

I’m upgrading Foreman 2.5 → 3.0 based on the directions at Foreman :: Manual . Steps 1 & 2 were successful.

But at Step 3 - Post-upgrade steps, and also Optional Step 3 (C) - Run foreman-installer, the commands are running into an error with Postgres. For example, several commands are failing with the error NameError: uninitialized constant Environment, shown below.

I searched for similar errors but only found https://community.theforeman.org/t/katello-upgrade-failing-on-db-migrate/24959/3. We do not use Katello. We only have one organization and location (See below).

Expected outcome:

I expected foreman-rake db:migrate to run without an error.

Foreman and Proxy versions:

  • Starting versions: Foreman 2.5.4
  • Ending version: The latest Foreman 3.0.x version

Smart Proxies include:
Logs, Puppet, HTTPBoot, DHCP, TFTP, Discovery, Registration, BMC, and Puppet CA

Foreman and Proxy plugin versions:

Before the upgrade:

foreman_discovery: 17.0.1
foreman_statistics: 1.1.1

Distribution and version:

Ubuntu 18.04.6 LTS

Other relevant data:

Foreman installer fails with these errors:

root@foreman:~# foreman-installer
2022-04-13 17:52:27 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-04-13 17:52:35 [NOTICE] [configure] Starting system configuration.
2022-04-13 17:52:45 [NOTICE] [configure] 250 configuration steps out of 1412 steps complete.
2022-04-13 17:52:46 [NOTICE] [configure] 500 configuration steps out of 1414 steps complete.
2022-04-13 17:52:50 [NOTICE] [configure] 750 configuration steps out of 1420 steps complete.
2022-04-13 17:52:50 [NOTICE] [configure] 1000 configuration steps out of 1521 steps complete.
2022-04-13 17:53:05 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-13 17:53:05 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-13 17:53:18 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-13 17:53:18 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-13 17:53:24 [NOTICE] [configure] 1250 configuration steps out of 1772 steps complete.
2022-04-13 17:53:25 [NOTICE] [configure] 1500 configuration steps out of 1973 steps complete.
2022-04-13 17:53:25 [NOTICE] [configure] 1750 configuration steps out of 2126 steps complete.
2022-04-13 17:53:43 [NOTICE] [configure] 2000 configuration steps out of 2175 steps complete.
2022-04-13 17:53:47 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/foreman.log
root@foreman:~#

And I suspect the db:migrate commands are failing because of this:

root@foreman:~# foreman-rake db:migrate
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
NameError: uninitialized constant Environment
/usr/share/foreman/vendor/ruby/2.5.0/gems/foreman_statistics-1.1.1/lib/foreman_statistics/engine.rb:91:in `block in <class:Engine>'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:428:in `instance_exec'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:428:in `block in make_lambda'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:605:in `block (2 levels) in default_terminator'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:604:in `catch'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:604:in `block in default_terminator'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:201:in `block in halting'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `block in invoke_before'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `invoke_before'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:134:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-6.0.3.7/lib/active_support/reloader.rb:88:in `prepare!'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/application/finisher.rb:113:in `block in <module:Finisher>'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:60:in `run_initializers'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/application.rb:363:in `initialize!'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/application.rb:339:in `require_environment!'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-6.0.3.7/lib/rails/application.rb:523:in `block in run_tasks_blocks'
Tasks: TOP => db:sessions:clear => environment
(See full trace by running task with --trace)
root@foreman:~#

The issue at Katello Upgrade failing on db:migrate - #3 by cflannigan suggests there is a mismatch in database for locations & organizations. That doesn’t seem to be the cause here. We only have one location & organization:

postgres=# \c foreman 
You are now connected to database "foreman" as user "postgres".
foreman=# select * from taxonomies;
 id |   name    |     type     |         created_at         |         updated_at         |      ignore_types      | ancestry |   title   | description 
----+-----------+--------------+----------------------------+----------------------------+------------------------+----------+-----------+-------------
  4 | ISG       | Organization | 2019-03-27 23:54:56.59449  | 2019-03-28 00:04:12.219474 | ---                   +|          | ISG       | 
    |           |              |                            |                            | - User                +|          |           | 
    |           |              |                            |                            | - SmartProxy          +|          |           | 
    |           |              |                            |                            | - Subnet              +|          |           | 
    |           |              |                            |                            | - ComputeResource     +|          |           | 
    |           |              |                            |                            | - Medium              +|          |           | 
    |           |              |                            |                            | - ProvisioningTemplate+|          |           | 
    |           |              |                            |                            | - Ptable              +|          |           | 
    |           |              |                            |                            | - Domain              +|          |           | 
    |           |              |                            |                            | - Realm               +|          |           | 
    |           |              |                            |                            | - Environment         +|          |           | 
    |           |              |                            |                            | - Hostgroup           +|          |           | 
    |           |              |                            |                            |                        |          |           | 
  3 | Wang Hall | Location     | 2019-03-27 23:54:38.379148 | 2019-03-28 00:04:49.216345 | ---                   +|          | Wang Hall | 
    |           |              |                            |                            | - User                +|          |           | 
    |           |              |                            |                            | - SmartProxy          +|          |           | 
    |           |              |                            |                            | - Subnet              +|          |           | 
    |           |              |                            |                            | - Medium              +|          |           | 
    |           |              |                            |                            | - ProvisioningTemplate+|          |           | 
    |           |              |                            |                            | - Ptable              +|          |           | 
    |           |              |                            |                            | - Domain              +|          |           | 
    |           |              |                            |                            | - Realm               +|          |           | 
    |           |              |                            |                            | - Environment         +|          |           | 
    |           |              |                            |                            | - Hostgroup           +|          |           | 
    |           |              |                            |                            |                        |          |           | 
(2 rows)

foreman=# 

I restored the VM snapshot back to before the upgrade, so we’re back online.

Is there anything I should change to Foreman before try the upgrade next time?

I do plan to upgrade Ubuntu 18.04 → 20.04, which will in turn upgrade the version of the Postgres DB. I’m unsure if that is related.

I finally fixed this.

The fix was apt remove ruby-foreman-statistics.

Why? Because that’s the code that contains the error, as shown in the log:

/usr/share/foreman/vendor/ruby/2.5.0/gems/foreman_statistics-1.1.1/lib/foreman_statistics/engine.rb:91:in `block in <class:Engine>'
1 Like