Upgrade to Foreman 3.11 failed with error

Problem:
By upgrading to Foreman 3.11 nad Katello to 4.13 i am getting an error:

2024-08-29 09:15:51 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2024-08-29 09:15:51 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2024-08-29 09:15:59 [NOTICE] [configure] 5000 configuration steps out of 5107 steps complete.
2024-08-29 09:16:11 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Exec resource 'foreman-rake-db:seed' failed. Logs:
  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]
    Adding autorequire relationship with User[foreman]
    Starting to evaluate the resource (4842 of 5107)
    '/usr/sbin/foreman-rake db:seed' won't be executed because of failed check 'refreshonly'
    Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
    '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
    Evaluated in 10.58 seconds
  Exec[foreman-rake-db:seed](provider=posix)
    Executing '/usr/sbin/foreman-rake db:seed'
  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns
    rake aborted!
    Foreman::PermissionMissingException: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["execute_jobs_on_infrastructure_hosts", "view_hosts", "view_remote_execution_features", "view_smart_proxies", "view_job_templates", "create_job_invocations", "view_job_invocations", "create_template_invocations", :view_job_templates, :view_job_invocations, :create_job_invocations, :create_template_invocations, :view_hosts, :view_smart_proxies, :view_remote_execution_features, :execute_jobs_on_infrastructure_hosts]
    /usr/share/foreman/app/models/role.rb:346:in `permission_records'
    /usr/share/foreman/app/models/role.rb:157:in `add_permissions'
    /usr/share/foreman/app/models/role.rb:210:in `add_permissions!'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:103:in `block (3 levels) in include_permissions_for_role'
    /usr/share/foreman/app/models/concerns/authorizable.rb:103:in `skip_permission_check'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:102:in `block (2 levels) in include_permissions_for_role'
    /usr/share/gems/gems/audited-5.6.0/lib/audited/auditor.rb:451:in `without_auditing'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:99:in `block in include_permissions_for_role'
    /usr/share/foreman/app/models/role.rb:239:in `ignore_locking'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:98:in `include_permissions_for_role'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:36:in `block in add_permissions_to_default_roles'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:32:in `each'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:32:in `add_permissions_to_default_roles'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:75:in `block (2 levels) in setup_roles!'
    /usr/share/gems/gems/audited-5.6.0/lib/audited/auditor.rb:451:in `without_auditing'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:59:in `block in setup_roles!'
    /usr/share/gems/gems/audited-5.6.0/lib/audited/auditor.rb:451:in `without_auditing'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:58:in `setup_roles!'
    /usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:46:in `setup!'
    /usr/share/foreman/app/registries/foreman/plugin.rb:310:in `finalize_setup!'
    /usr/share/foreman/config/initializers/foreman.rb:28:in `block (2 levels) in <top (required)>'
    /usr/share/foreman/config/initializers/foreman.rb:27:in `each'
    /usr/share/foreman/config/initializers/foreman.rb:27:in `block in <top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/lazy_load_hooks.rb:68:in `block in execute_hook'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/lazy_load_hooks.rb:51:in `each'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application/finisher.rb:140:in `block in <module:Finisher>'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.14/lib/zeitwerk/kernel.rb:34:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:seed => db:load_config => environment
    (See full trace by running task with --trace)

1 error was detected during installation.

Expected outcome:

Foreman and Proxy versions:
Foreman foreman-3.10.0
Katello-4.12.1
Foreman and Proxy plugin versions:

Distribution and version:
RHEL 8.10
Other relevant data:

Can someone please help? Does anyone have a solution for this?

Fixes #37503 - Delay plugin finalization until seeds are seeded by adamruzicka · Pull Request #10182 · theforeman/foreman · GitHub should fix this, but it was not yet backported to 3.11.

You can manually apply the patch and see if it helps?

Same issue on Ubuntu 22.04 without Katello upgrading 3.11->3.11.1. I ran normal system updates then “foreman-installer”. This failed with “FATAL: role “foreman” does not exist”. I confirmed the database and role still exist. Then I tried “sudo foreman-rake db:migrate”. After that foreman-rake-db:seed began failing with the same error as OP. Here are all the package updates that installed before:

Start-Date: 2024-08-30  14:37:45
Commandline: apt dist-upgrade -y
Requested-By: localadmin (1000)
Upgrade: libext2fs2:amd64 (1.46.5-2ubuntu1.1, 1.46.5-2ubuntu1.2), libpython3-dev:amd64 (3.10.6-1~22.04, 3.10.6-1~22.04.1), libldap-common:amd64 (2.5.18+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.2), python3-minimal:amd64 (3.10.6-1~22.04, 3.10.6-1~22.04.1), slapd:amd64 (2.5.18+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.2), libldap-dev:amd64 (2.5.18+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.2), foreman-installer:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), python3:amd64 (3.10.6-1~22.04, 3.10.6-1~22.04.1), foreman-redis:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), ldap-utils:amd64 (2.5.18+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.2), foreman-postgresql:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), libldap-2.5-0:amd64 (2.5.18+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.2), libcom-err2:amd64 (1.46.5-2ubuntu1.1, 1.46.5-2ubuntu1.2), foreman-cli:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), ruby-foreman-puppet:amd64 (6.3.0-1, 7.0.0-1~fm3.11), python3-dev:amd64 (3.10.6-1~22.04, 3.10.6-1~22.04.1), foreman-service:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), libpython3-stdlib:amd64 (3.10.6-1~22.04, 3.10.6-1~22.04.1), foreman-proxy:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), ubuntu-standard:amd64 (1.481.2, 1.481.3), puppetserver:amd64 (7.17.1-1jammy, 7.17.2-1jammy), python3-apport:amd64 (2.20.11-0ubuntu82.5, 2.20.11-0ubuntu82.6), foreman:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), libldap2-dev:amd64 (2.5.18+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.2), ubuntu-server:amd64 (1.481.2, 1.481.3), logsave:amd64 (1.46.5-2ubuntu1.1, 1.46.5-2ubuntu1.2), foreman-dynflow-sidekiq:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), foreman-debug:amd64 (3.11.0-1+ubuntu2204, 3.11.1-1+ubuntu2204), python3-problem-report:amd64 (2.20.11-0ubuntu82.5, 2.20.11-0ubuntu82.6), libss2:amd64 (1.46.5-2ubuntu1.1, 1.46.5-2ubuntu1.2), apport:amd64 (2.20.11-0ubuntu82.5, 2.20.11-0ubuntu82.6), ubuntu-minimal:amd64 (1.481.2, 1.481.3), puppet-agent:amd64 (7.31.0-1jammy, 7.32.1-1jammy), e2fsprogs:amd64 (1.46.5-2ubuntu1.1, 1.46.5-2ubuntu1.2)
End-Date: 2024-08-30  14:45:15