Foreman-rake db: migrate aborted! NameError: uninitialized constant Setting::BLANK_ATTRS

Problem:
I have taken over a server running Foreman 2.5 on Ubuntu 20.04
Encountered only minor issues during the updates to 3.5
However after the update to Foreman 3.6 I receive a abort error during the foreman-rake db:migrate.

I’ve tried:
sudo gem update bundler
sudo gem update --system

My initial thinking is there is some issue with environment variables and I’m completely new to foreman. Any ideas to point me in the right direction would be awesome.

Expected outcome:
Expected same result as updates 3.0 - 3.5. I expected no errors

Foreman and Proxy versions:
Foreman 3.6

Distribution and version:
Ubuntu 20.04

Other relevant data:
root@foreman:/tmp# foreman-rake db:migrate --trace
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
NameError: uninitialized constant Setting::BLANK_ATTRS
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_ansible-6.3.1/app/models/setting/ansible.rb:6:in <class:Ansible>' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_ansible-6.3.1/app/models/setting/ansible.rb:5:in class:Setting
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_ansible-6.3.1/app/models/setting/ansible.rb:3:in <top (required)>' /usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require’
/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies/interlock.rb:14:in block in loading’
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/concurrency/share_lock.rb:151:in exclusive' /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies/interlock.rb:13:in loading’
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_ansible-6.3.1/lib/foreman_ansible/engine.rb:25:in block in <class:Engine>' /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/initializable.rb:32:in instance_exec’
/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/initializable.rb:32:in run' /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/initializable.rb:61:in block in run_initializers’
/usr/lib/ruby/2.7.0/tsort.rb:228:in block in tsort_each' /usr/lib/ruby/2.7.0/tsort.rb:350:in block (2 levels) in each_strongly_connected_component’
/usr/lib/ruby/2.7.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from' /usr/lib/ruby/2.7.0/tsort.rb:431:in each_strongly_connected_component_from’
/usr/lib/ruby/2.7.0/tsort.rb:421:in block in each_strongly_connected_component_from' /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/initializable.rb:50:in each’
/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/initializable.rb:50:in tsort_each_child' /usr/lib/ruby/2.7.0/tsort.rb:415:in call’
/usr/lib/ruby/2.7.0/tsort.rb:415:in each_strongly_connected_component_from' /usr/lib/ruby/2.7.0/tsort.rb:349:in block in each_strongly_connected_component’
/usr/lib/ruby/2.7.0/tsort.rb:347:in each' /usr/lib/ruby/2.7.0/tsort.rb:347:in call’
/usr/lib/ruby/2.7.0/tsort.rb:347:in each_strongly_connected_component' /usr/lib/ruby/2.7.0/tsort.rb:226:in tsort_each’
/usr/lib/ruby/2.7.0/tsort.rb:205:in tsort_each' /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/initializable.rb:60:in run_initializers’
/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/application.rb:391:in initialize!' /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in public_send’
/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in method_missing' /usr/share/foreman/config/environment.rb:5:in <top (required)>’
/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' /usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require’
/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/application.rb:367:in require_environment!' /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/application.rb:533:in block in run_tasks_blocks’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in block in execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in each’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in block in invoke_with_call_chain’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:243:in block in invoke_prerequisites' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in each’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in invoke_prerequisites' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:218:in block in invoke_with_call_chain’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:243:in block in invoke_prerequisites' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in each’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in invoke_prerequisites' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:218:in block in invoke_with_call_chain’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in invoke' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in invoke_task’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block (2 levels) in top_level' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in each’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block in top_level' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in run_with_threads’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in top_level' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in block in run’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in standard_exception_handling' /usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in run’
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>' /usr/bin/rake:23:in load’
/usr/bin/rake:23:in `’
Tasks: TOP => db:migrate => db:load_config => environment
root@foreman:/tmp#

Just an educated guess, but looks like you are running an old version of the Foreman Ansible Plugin (prior to Fixes #32409 - Use new DSL to define settings by adiabramovitch · Pull Request #448 · theforeman/foreman_ansible · GitHub, which was merged in October 2021).
I am not familiar with Foreman setups on Ubuntu, but could it be you missed changing the the foreman-plugins repository to newer versions during your upgrades?

Thanks for the quick reply!

I might need to try to install or reinstall them individually but my repo is as follows:

deb http://deb.theforeman.org/ focal 3.6
deb http://deb.theforeman.org/ plugins 3.6

I’ll take a second look.

Are you sure the packages were properly updated?

The trace shows foreman_ansible 6.3.1, which is from Foreman 2.5, the current version in 3.6 is 11.1.2.

Can you post the output of apt policy ruby-foreman-ansible please?

Our packaging has its own vendoring and this shouldn’t be needed. It can in fact break things in unexpected ways.

I am away from that system at the moment and can’t post the output but you are correct. When I checked the ansible version was still at 6.3.1. It looks like somewhere along the way these packages were not updated.

To wrap things up here I was hoping to get some clarification on how the “packaging has its own vendoring”. I am assuming they need to be managed through the foreman-installer because messing with them independently would break the “hooks” into foreman? Would you mind elaborating on this?

Lastly the manual states that when upgrading from 2.5 to 3.0 “if you wish to continue using Puppet:”

apt-get install ruby-foreman-puppet

But none of the other versions have this step. On a fresh install of 3.0 or higher, is Puppet part of the default foreman package? I have read a few things about how the Foreman/Puppet infrastructure changes from 2.5 to 3.0 but I’m still a bit confused by this and how that effects the upgrade process?