Issues during upgrade 1.18.2 to 1.19, following the upgrade guide

Support
1

Problem:
During the upgrade of 1.18.2, following the guide provided on Foreman :: Manual, I had 2 errors :

  • SQLite was introduced, but package was not installed
  • Syntax error on TRUNCATE during db:sessions:clear

The first issue I solved, but indicates a missing additional command in the upgrade guide.
The second issue was the reason to stop the upgrade to 1.19 and rollback to 1.18.2
A full overview of command given, errors seen can be found in the “Other relevant info”

Expected outcome:
The upgrade guide should be reviewed and additional things should be added.
Things missing as far as I can see :

  • Installation of foreman-sqlite3
  • Some magic to get around the message “API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.” (probably an additional foreman-rake command to run…

Foreman and Proxy versions:
Started from foreman 1.18.2

Foreman and Proxy plugin versions:

Other relevant data:
The server on my side is running Ubuntu 16.04 and was once installed using the foreman-installer on foreman version 1.15.
Complete overview of steps taken for this upgrade :

  1. Updated /etc/apt/sources.list.d/foreman.list with new version numbers
  2. Stop apache2 service
  3. Run “sudo apt update”
  4. Run “sudo apt-get --only-upgrade install ruby* foreman*”
    During this step a warning was given there is a difference between the installed version of /etc/foreman/database.yml and the version in the package :

— /etc/foreman/database.yml 2018-09-17 08:53:39.645978212 +0200
+++ /etc/foreman/database.yml.dpkg-new 2018-08-31 02:40:09.000000000 +0200
@@ -1,10 +1,22 @@
-### File managed with puppet ###
-## Module: ‘foreman’
+# SQLite version 3.x
+# gem install sqlite3
+development:

  • adapter: sqlite3
  • database: db/development.sqlite3
  • pool: 10
  • timeout: 5000

+# Warning: The database defined as “test” will be erased and
+# re-generated from your development database when you run “rake”.
+# Do not set this db to the same as development or production.
+test:

  • adapter: sqlite3
  • database: db/test.sqlite3
  • pool: 10
  • timeout: 5000

-# Database is managed by foreman::database::postgresql
production:

  • adapter: postgresql
  • database: foreman
  • username: foreman
  • password: “XkNykU934UwtbappwTtjnyDrjUA2TC88”
  • pool: 5
  • adapter: sqlite3
  • database: db/production.sqlite3
  • pool: 10
  • timeout: 5000
  1. Run “sudo foreman-rake db:migrate”
    At this step, the following error occured :

rake aborted!
Gem::LoadError: Specified ‘sqlite3’ for database adapter, but the gem is not loaded. Add gem 'sqlite3' to your Gemfile (and ensure its version is at the minimum required by ActiveRecord).
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/connection_specification.rb:188:in rescue in spec' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/connection_specification.rb:185:inspec’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:880:in establish_connection' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_handling.rb:58:inestablish_connection’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/railtie.rb:124:in block (2 levels) in <class:Railtie>' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:69:ininstance_eval’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:69:in block in execute_hook' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:60:inwith_execution_control’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:65:in execute_hook' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:41:inblock in on_load’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:40:in each' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:40:inon_load’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/railtie.rb:120:in block in <class:Railtie>' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:30:ininstance_exec’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:30:in run' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:59:inblock in run_initializers’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:58:in run_initializers' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/application.rb:353:ininitialize!’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/railtie.rb:185:in public_send' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/railtie.rb:185:inmethod_missing’
/usr/share/foreman/config/environment.rb:5:in <top (required)>' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:inrequire’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:in block in require' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:258:inload_dependency’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:in require' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/application.rb:329:inrequire_environment!’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/application.rb:445:in `block in run_tasks_blocks’

Caused by:
Gem::LoadError: sqlite3 is not part of the bundle. Add it to Gemfile.
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:10:in <top (required)>' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:inrequire’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:in block in require' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:258:inload_dependency’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:in require' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/connection_specification.rb:186:inspec’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:880:in establish_connection' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_handling.rb:58:inestablish_connection’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/railtie.rb:124:in block (2 levels) in <class:Railtie>' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:69:ininstance_eval’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:69:in block in execute_hook' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:60:inwith_execution_control’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:65:in execute_hook' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:41:inblock in on_load’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:40:in each' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/lazy_load_hooks.rb:40:inon_load’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/railtie.rb:120:in block in <class:Railtie>' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:30:ininstance_exec’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:30:in run' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:59:inblock in run_initializers’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/initializable.rb:58:in run_initializers' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/application.rb:353:ininitialize!’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/railtie.rb:185:in public_send' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/railtie.rb:185:inmethod_missing’
/usr/share/foreman/config/environment.rb:5:in <top (required)>' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:inrequire’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:in block in require' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:258:inload_dependency’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:292:in require' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/application.rb:329:inrequire_environment!’
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/application.rb:445:in `block in run_tasks_blocks’
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)

This was solved by taking an additional step, which is not documented in the upgrade guide :
6. sudo apt install foreman-sqlite3
7. Rerun db:migrate
Now the following messages shows up :

API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.

This message was ignored for the moment.
8. Running “sudo foreman-rake db:seed”, this showed the same warning on the API controllers, also ignored in this case.
9. Running “sudo foreman-rake tmp:cache:clear”
10. Running “sudo foreman-rake db:sessions:clear”
This resulted in the following error :

API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
ActiveRecord::StatementInvalid: SQLite3::SQLException: near “TRUNCATE”: syntax error: TRUNCATE TABLE sessions
/usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:in initialize' /usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:innew’
/usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:in prepare' /usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:137:inexecute’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:251:in block (2 levels) in execute' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:46:inblock in permit_concurrent_loads’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:185:in yield_shares' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:45:inpermit_concurrent_loads’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:250:in block in execute' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:613:inblock (2 levels) in log’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:612:in block in log' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:21:ininstrument’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:604:in log' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:249:inexecute’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-session_store-1.1.1/lib/tasks/database.rake:12:in `block (2 levels) in <top (required)>’

Caused by:
SQLite3::SQLException: near “TRUNCATE”: syntax error
/usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:in initialize' /usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:innew’
/usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:in prepare' /usr/share/foreman/vendor/ruby/2.3.0/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:137:inexecute’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:251:in block (2 levels) in execute' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:46:inblock in permit_concurrent_loads’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:185:in yield_shares' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:45:inpermit_concurrent_loads’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:250:in block in execute' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:613:inblock (2 levels) in log’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:612:in block in log' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:21:ininstrument’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:604:in log' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:249:inexecute’
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-session_store-1.1.1/lib/tasks/database.rake:12:in `block (2 levels) in <top (required)>’
Tasks: TOP => db:sessions:clear
(See full trace by running task with --trace)

At this point I decided to rollback the snapshot I created and stop the upgrade…

2

It looks to me, that the deb package has overridden your database.yml config file, which is normally managed by foreman-installer. That would be debian packaging bug, cc @mmoll mayne he would know more.

That file contains DB connection info. Before you move on, make sure you revert the change. Yoi still want to use the original MySQL DB instead of new sqlite, you’d lose all data. That also addresses the missing dep, in fact you don’t need sqlite if you’re on MySQL.

Another option is to rerun installer, which will reconfigure everything. IIRC correctly, we recommend this step in upgrade notes. Though it’s not ideal to override the config by package and then override it back with installer.

3

Yes, that file should not get replaced with the default when updating. Re-Running the installer should fix this, however I have no idea if you modified config files outside the installer setup (these modifications will get overwritten).

4

Thanks for the responses.
The rerun of the installer was a step I did not yet take since this is part of the upgrade step 3C.
Does this also mean that, when the system was installed using the foreman-installer, step 3A should be skipped and only step 3C is needed ?

I did a new upgrade, but now with some different decisions.
In the apt-get --only-upgrade step, I did not replace any configs with the package versions, leaving the following files untouched (answer “N”) :

  • /etc/foreman/database.yml
  • /etc/foreman-proxy/settings.d/puppetca.yml

After this, I skipped step 3-A and did run the foreman-installer from step 3-C
Result was an upgraded system on version 1.19.0.

So, as a suggestion on the upgrade guide.
Maybe note that the suggested config file changes in the apt-get install flow should not be granted (answering “N” to leave the original files in place) in case the system was installed using foreman-installer.
And step 3-A might also be skipped in case foreman-installer was used, as long as step 3-C is taking its place

For the rest… great work on the upgrade… life is easy this way :wink:

5

Please open a pull request to https://github.com/theforeman/theforeman.org :+1: