Foreman-install to upgrade to foreman 3.8 fails (foreman-rake db:seed)

Problem: When trying to upgrade to foreman 3.8 and Katello 4.10, foreman-rake db:seed fails with a ArgumentError: unknown keyword: :value error. Also, we are assuming this is related, but sidekiq is using 100% cpu utilization

Expected outcome: Foreman will upgrade successfully

Foreman and Proxy versions: 3.7&4.9 to 3.8&4.10

Foreman and Proxy plugin versions: Disabled all plugins except for puppet while diagnosing.

Distribution and version: Rocky 8

Other relevant data:

Puppet plugin is version 6.0.1 (the latest available from dnf)

I also have katello, and no independent smart proxies

Can you please post the whole stacktrace that is printed when you run foreman-rake db:seed?

Error 1: Puppet Exec resource 'foreman-rake-db:seed' failed. Logs:
  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]
    Adding autorequire relationship with User[foreman]
    Starting to evaluate the resource (1583 of 1645)
    '/usr/sbin/foreman-rake db:seed' won't be executed because of failed check 'refreshonly'
    Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
    '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
    Evaluated in 12.27 seconds
  Exec[foreman-rake-db:seed](provider=posix)
    Executing '/usr/sbin/foreman-rake db:seed'
  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns
    rake aborted!
    ArgumentError: unknown keyword: :value
    /usr/share/foreman/app/services/setting_registry.rb:151:in `_add'
    /usr/share/foreman/app/models/setting.rb:198:in `block in load_defaults'
    /usr/share/foreman/app/models/setting.rb:195:in `each'
    /usr/share/foreman/app/models/setting.rb:195:in `load_defaults'
    /usr/share/gems/gems/foreman_git_templates-1.0.6/app/models/setting/git_templates.rb:13:in `load_defaults'
    /usr/share/foreman/app/services/setting_registry.rb:114:in `each'
    /usr/share/foreman/app/services/setting_registry.rb:114:in `load'
    /usr/share/foreman/config/initializers/foreman.rb:37:in `block in <top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:427:in `instance_exec'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:427:in `block in make_lambda'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:603:in `catch'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:603:in `block in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:199:in `block in halting'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:512:in `block in invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:512:in `each'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:512:in `invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:105:in `run_callbacks'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/reloader.rb:88:in `prepare!'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.11/lib/zeitwerk/kernel.rb:38:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.6/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:seed => db:load_config => environment
    (See full trace by running task with --trace)

your stacktrace indicates that you have foreman_git_templates installed, and that is known not to be compatible with recent versions of Foreman. You can try applying the linked patch, or removing the plugin and retrying the upgrade.

Thank you. I had disabled it using foreman-installer -i and didn’t realize that didn’t actually remove it. When I uninstalled it using DNF, it worked perfectly.