Upgrading from foreman-3.0.1-1 to foreman-3.1.0-1 is failing

Problem:
Upgrading from foreman-3.0.1-1 to foreman-3.1.0-1. Dynflow-sidekiq services not being able to start during and after running foreman-installer.

Expected outcome:
A working installation

Foreman and Proxy versions:
foreman-3.1.0-1.el7
foreman-proxy-3.1.0-1.el7

Foreman and Proxy plugin versions:
foreman-plugin-3.1.0-1.el7

Distribution and version:
CentOS 7

Other relevant data:
Foreman-installer (attempt 1)

[root@katello01 ~]# foreman-installer
2022-01-11 11:39:19 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-01-11 11:39:24 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-01-11 11:39:24 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-01-11 11:39:29 [NOTICE] [configure] Starting system configuration.
2022-01-11 11:39:40 [NOTICE] [configure] 250 configuration steps out of 2222 steps complete.
2022-01-11 11:39:47 [NOTICE] [configure] 500 configuration steps out of 2222 steps complete.
2022-01-11 11:39:50 [NOTICE] [configure] 750 configuration steps out of 2228 steps complete.
2022-01-11 11:39:54 [NOTICE] [configure] 1000 configuration steps out of 2237 steps complete.
2022-01-11 11:40:03 [NOTICE] [configure] 1500 configuration steps out of 2257 steps complete.
2022-01-11 11:46:53 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2022-01-11 11:46:53 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2022-01-11 11:46:54 [NOTICE] [configure] 1750 configuration steps out of 2423 steps complete.
2022-01-11 11:46:54 [NOTICE] [configure] 2000 configuration steps out of 2592 steps complete.
2022-01-11 11:46:54 [NOTICE] [configure] 2250 configuration steps out of 2767 steps complete.
2022-01-11 11:47:06 [NOTICE] [configure] 2500 configuration steps out of 2800 steps complete.
2022-01-11 11:47:27 [NOTICE] [configure] 2750 configuration steps out of 2800 steps complete.
2022-01-11 11:47:34 [NOTICE] [configure] System configuration has finished.

Forman-installer (attempt 2)

[root@katello01 ~]# foreman-installer
2022-01-11 11:49:58 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-01-11 11:50:04 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-01-11 11:50:04 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-01-11 11:50:08 [NOTICE] [configure] Starting system configuration.
2022-01-11 11:50:20 [NOTICE] [configure] 250 configuration steps out of 2222 steps complete.
2022-01-11 11:50:23 [NOTICE] [configure] 500 configuration steps out of 2222 steps complete.
2022-01-11 11:50:25 [NOTICE] [configure] 750 configuration steps out of 2228 steps complete.
2022-01-11 11:50:27 [NOTICE] [configure] 1000 configuration steps out of 2237 steps complete.
2022-01-11 11:50:28 [NOTICE] [configure] 1250 configuration steps out of 2242 steps complete.
2022-01-11 11:50:29 [NOTICE] [configure] 1500 configuration steps out of 2243 steps complete.
2022-01-11 11:51:10 [NOTICE] [configure] 1750 configuration steps out of 2409 steps complete.
2022-01-11 11:51:10 [NOTICE] [configure] 2000 configuration steps out of 2578 steps complete.
2022-01-11 11:51:11 [NOTICE] [configure] 2250 configuration steps out of 2786 steps complete.
2022-01-11 11:51:11 [NOTICE] [configure] 2500 configuration steps out of 2786 steps complete.
2022-01-11 11:51:31 [ERROR ] [configure] Systemd start for dynflow-sidekiq@orchestrator failed!
2022-01-11 11:51:31 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2022-01-11 11:51:31 [ERROR ] [configure] -- Logs begin at Tue 2022-01-11 11:27:28 CET, end at Tue 2022-01-11 11:51:31 CET. --
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:11 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.333Z 28197 TID-kvl INFO: GitLab reliable fetch activated!
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.346Z 28197 TID-kul INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-28197", :url=>"redis://localhost:6379/0"}
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:16 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:27 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: User with login admin already exists, not seeding as admin.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 390 extra bytes after the deserialized object
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `block in find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `map'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence.rb:60:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/jobs/application_job.rb:11:in `spawn_if_missing'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
2022-01-11 11:51:31 [ERROR ] [configure] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[orchestrator]/Service[dynflow-sidekiq@orchestrator]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@orchestrator failed!
2022-01-11 11:51:31 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2022-01-11 11:51:31 [ERROR ] [configure] -- Logs begin at Tue 2022-01-11 11:27:28 CET, end at Tue 2022-01-11 11:51:31 CET. --
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:11 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.333Z 28197 TID-kvl INFO: GitLab reliable fetch activated!
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.346Z 28197 TID-kul INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-28197", :url=>"redis://localhost:6379/0"}
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:16 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:27 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: User with login admin already exists, not seeding as admin.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 390 extra bytes after the deserialized object
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `block in find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `map'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence.rb:60:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/jobs/application_job.rb:11:in `spawn_if_missing'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
2022-01-11 11:52:15 [NOTICE] [configure] 2750 configuration steps out of 2786 steps complete.
2022-01-11 11:52:19 [NOTICE] [configure] System configuration has finished.

Dynflow-sidekiq@orchestrator LOG

-- Logs begin at Tue 2022-01-11 11:27:28 CET, end at Tue 2022-01-11 11:56:05 CET. --
Jan 11 11:27:35 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:27:41 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: 2022-01-11T10:27:41.498Z 1694 TID-7e INFO: GitLab reliable fetch activated!
Jan 11 11:27:41 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: 2022-01-11T10:27:41.498Z 1694 TID-8m INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-1694", :url=>"redis://localhost:6379/0"}
Jan 11 11:27:48 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:27:53 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-1.0.5/lib/foreman_puppet/register.rb:141: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:27:53 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
Jan 11 11:28:39 katello01.infra.aza.nu systemd[1]: Started Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:33:04 katello01.infra.aza.nu systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:33:06 katello01.infra.aza.nu systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:33:06 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:33:07 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: 2022-01-11T10:33:07.315Z 16243 TID-b8n INFO: GitLab reliable fetch activated!
Jan 11 11:33:07 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: 2022-01-11T10:33:07.315Z 16243 TID-b8b INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-16243", :url=>"redis://localhost:6379/0"}
Jan 11 11:33:10 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:33:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:33:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: DETAIL:  Key (typname, typnamespace)=(dynflow_output_chunks_id_seq, 2200) already exists.
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:500:in `_execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block (2 levels) in execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block in execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:88:in `hold'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/query.rb:50:in `execute_dui'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/query.rb:43:in `execute_ddl'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/schema_methods.rb:690:in `create_table_from_generator'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/schema_methods.rb:194:in `create_table'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/shared/postgres.rb:459:in `create_table'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel_migrations/021_create_output_chunks.rb:5:in `block (2 levels) in <top (required)>'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:110:in `instance_exec'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:110:in `apply'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:563:in `block (2 levels) in run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/transactions.rb:258:in `_transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/transactions.rb:233:in `block in transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/transactions.rb:195:in `transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:479:in `checked_transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:562:in `block in run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:559:in `each'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:559:in `run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:406:in `run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:271:in `migrate_db'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:59:in `initialize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:200:in `new'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:200:in `initialize_persistence'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:142:in `block (2 levels) in world_config'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/config.rb:43:in `method_missing'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/world.rb:29:in `initialize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:79:in `new'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:79:in `initialize_world'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:105:in `init_world'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
...skipping...
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
Jan 11 11:55:38 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service holdoff time over, scheduling restart.
Jan 11 11:55:38 katello01.infra.aza.nu systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:38 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:55:39 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: 2022-01-11T10:55:39.598Z 29462 TID-n0i INFO: GitLab reliable fetch activated!
Jan 11 11:55:39 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: 2022-01-11T10:55:39.599Z 29462 TID-n26 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-29462", :url=>"redis://localhost:6379/0"}
Jan 11 11:55:42 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:55:46 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:55:46 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
Jan 11 11:55:53 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: User with login admin already exists, not seeding as admin.
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: 390 extra bytes after the deserialized object
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `block in find_execution_plans'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `map'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `find_execution_plans'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence.rb:60:in `find_execution_plans'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/app/jobs/application_job.rb:11:in `spawn_if_missing'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
Jan 11 11:55:58 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service holdoff time over, scheduling restart.
Jan 11 11:55:58 katello01.infra.aza.nu systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:58 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:56:00 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: 2022-01-11T10:56:00.110Z 29539 TID-n3r INFO: GitLab reliable fetch activated!
Jan 11 11:56:00 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: 2022-01-11T10:56:00.111Z 29539 TID-n2j INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-29539", :url=>"redis://localhost:6379/0"}
Jan 11 11:56:03 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:56:06 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:56:06 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here

Hey @cazzen2k4
Sorry to hear you’re having issues.
Did you get any further?

Hi @mcorr

Unfortunately not. I’ve taken a snapshot of the server so I can revert to before the upgrade. I’ve tried the upgrade multiple times but without any success.

This log-output sticks out to me looking at the dynflow-sidekiq@orchestrator log above.

an 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: DETAIL:  Key (typname, typnamespace)=(dynflow_output_chunks_id_seq, 2200) already exists.

I’ll try and get some eyes on this !

Can anyone from @katello advise as to what the next steps to troubleshoot this might be?

Please try clearing out ALL your tasks first:

1 Like