Upgrading from foreman-3.0.1-1 to foreman-3.1.0-1 is failing

Support
, ,
1

Problem:
Upgrading from foreman-3.0.1-1 to foreman-3.1.0-1. Dynflow-sidekiq services not being able to start during and after running foreman-installer.

Expected outcome:
A working installation

Foreman and Proxy versions:
foreman-3.1.0-1.el7
foreman-proxy-3.1.0-1.el7

Foreman and Proxy plugin versions:
foreman-plugin-3.1.0-1.el7

Distribution and version:
CentOS 7

Other relevant data:
Foreman-installer (attempt 1)

[root@katello01 ~]# foreman-installer
2022-01-11 11:39:19 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-01-11 11:39:24 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-01-11 11:39:24 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-01-11 11:39:29 [NOTICE] [configure] Starting system configuration.
2022-01-11 11:39:40 [NOTICE] [configure] 250 configuration steps out of 2222 steps complete.
2022-01-11 11:39:47 [NOTICE] [configure] 500 configuration steps out of 2222 steps complete.
2022-01-11 11:39:50 [NOTICE] [configure] 750 configuration steps out of 2228 steps complete.
2022-01-11 11:39:54 [NOTICE] [configure] 1000 configuration steps out of 2237 steps complete.
2022-01-11 11:40:03 [NOTICE] [configure] 1500 configuration steps out of 2257 steps complete.
2022-01-11 11:46:53 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2022-01-11 11:46:53 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2022-01-11 11:46:54 [NOTICE] [configure] 1750 configuration steps out of 2423 steps complete.
2022-01-11 11:46:54 [NOTICE] [configure] 2000 configuration steps out of 2592 steps complete.
2022-01-11 11:46:54 [NOTICE] [configure] 2250 configuration steps out of 2767 steps complete.
2022-01-11 11:47:06 [NOTICE] [configure] 2500 configuration steps out of 2800 steps complete.
2022-01-11 11:47:27 [NOTICE] [configure] 2750 configuration steps out of 2800 steps complete.
2022-01-11 11:47:34 [NOTICE] [configure] System configuration has finished.

Forman-installer (attempt 2)

[root@katello01 ~]# foreman-installer
2022-01-11 11:49:58 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-01-11 11:50:04 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-01-11 11:50:04 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-01-11 11:50:08 [NOTICE] [configure] Starting system configuration.
2022-01-11 11:50:20 [NOTICE] [configure] 250 configuration steps out of 2222 steps complete.
2022-01-11 11:50:23 [NOTICE] [configure] 500 configuration steps out of 2222 steps complete.
2022-01-11 11:50:25 [NOTICE] [configure] 750 configuration steps out of 2228 steps complete.
2022-01-11 11:50:27 [NOTICE] [configure] 1000 configuration steps out of 2237 steps complete.
2022-01-11 11:50:28 [NOTICE] [configure] 1250 configuration steps out of 2242 steps complete.
2022-01-11 11:50:29 [NOTICE] [configure] 1500 configuration steps out of 2243 steps complete.
2022-01-11 11:51:10 [NOTICE] [configure] 1750 configuration steps out of 2409 steps complete.
2022-01-11 11:51:10 [NOTICE] [configure] 2000 configuration steps out of 2578 steps complete.
2022-01-11 11:51:11 [NOTICE] [configure] 2250 configuration steps out of 2786 steps complete.
2022-01-11 11:51:11 [NOTICE] [configure] 2500 configuration steps out of 2786 steps complete.
2022-01-11 11:51:31 [ERROR ] [configure] Systemd start for dynflow-sidekiq@orchestrator failed!
2022-01-11 11:51:31 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2022-01-11 11:51:31 [ERROR ] [configure] -- Logs begin at Tue 2022-01-11 11:27:28 CET, end at Tue 2022-01-11 11:51:31 CET. --
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:11 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.333Z 28197 TID-kvl INFO: GitLab reliable fetch activated!
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.346Z 28197 TID-kul INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-28197", :url=>"redis://localhost:6379/0"}
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:16 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:27 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: User with login admin already exists, not seeding as admin.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 390 extra bytes after the deserialized object
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `block in find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `map'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence.rb:60:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/jobs/application_job.rb:11:in `spawn_if_missing'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
2022-01-11 11:51:31 [ERROR ] [configure] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[orchestrator]/Service[dynflow-sidekiq@orchestrator]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@orchestrator failed!
2022-01-11 11:51:31 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2022-01-11 11:51:31 [ERROR ] [configure] -- Logs begin at Tue 2022-01-11 11:27:28 CET, end at Tue 2022-01-11 11:51:31 CET. --
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:11 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.333Z 28197 TID-kvl INFO: GitLab reliable fetch activated!
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 2022-01-11T10:51:13.346Z 28197 TID-kul INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-28197", :url=>"redis://localhost:6379/0"}
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:16 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:20 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:27 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: User with login admin already exists, not seeding as admin.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: 390 extra bytes after the deserialized object
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `block in find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `map'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence.rb:60:in `find_execution_plans'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/app/jobs/application_job.rb:11:in `spawn_if_missing'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[28197]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
2022-01-11 11:51:31 [ERROR ] [configure] Jan 11 11:51:31 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
2022-01-11 11:52:15 [NOTICE] [configure] 2750 configuration steps out of 2786 steps complete.
2022-01-11 11:52:19 [NOTICE] [configure] System configuration has finished.

Dynflow-sidekiq@orchestrator LOG

-- Logs begin at Tue 2022-01-11 11:27:28 CET, end at Tue 2022-01-11 11:56:05 CET. --
Jan 11 11:27:35 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:27:41 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: 2022-01-11T10:27:41.498Z 1694 TID-7e INFO: GitLab reliable fetch activated!
Jan 11 11:27:41 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: 2022-01-11T10:27:41.498Z 1694 TID-8m INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-1694", :url=>"redis://localhost:6379/0"}
Jan 11 11:27:48 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:27:53 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-1.0.5/lib/foreman_puppet/register.rb:141: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:27:53 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[1694]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
Jan 11 11:28:39 katello01.infra.aza.nu systemd[1]: Started Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:33:04 katello01.infra.aza.nu systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:33:06 katello01.infra.aza.nu systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:33:06 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:33:07 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: 2022-01-11T10:33:07.315Z 16243 TID-b8n INFO: GitLab reliable fetch activated!
Jan 11 11:33:07 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: 2022-01-11T10:33:07.315Z 16243 TID-b8b INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-16243", :url=>"redis://localhost:6379/0"}
Jan 11 11:33:10 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:33:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:33:13 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: DETAIL:  Key (typname, typnamespace)=(dynflow_output_chunks_id_seq, 2200) already exists.
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:500:in `_execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block (2 levels) in execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block in execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:88:in `hold'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `execute'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/query.rb:50:in `execute_dui'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/query.rb:43:in `execute_ddl'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/schema_methods.rb:690:in `create_table_from_generator'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/schema_methods.rb:194:in `create_table'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/shared/postgres.rb:459:in `create_table'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel_migrations/021_create_output_chunks.rb:5:in `block (2 levels) in <top (required)>'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:110:in `instance_exec'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:110:in `apply'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:563:in `block (2 levels) in run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/transactions.rb:258:in `_transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/transactions.rb:233:in `block in transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/transactions.rb:195:in `transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:479:in `checked_transaction'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:562:in `block in run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:559:in `each'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:559:in `run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/extensions/migration.rb:406:in `run'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:271:in `migrate_db'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:59:in `initialize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:200:in `new'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:200:in `initialize_persistence'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:142:in `block (2 levels) in world_config'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/config.rb:43:in `method_missing'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/world.rb:29:in `initialize'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:79:in `new'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:79:in `initialize_world'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:105:in `init_world'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
...skipping...
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
Jan 11 11:55:37 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29391]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
Jan 11 11:55:37 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
Jan 11 11:55:38 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service holdoff time over, scheduling restart.
Jan 11 11:55:38 katello01.infra.aza.nu systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:38 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:55:39 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: 2022-01-11T10:55:39.598Z 29462 TID-n0i INFO: GitLab reliable fetch activated!
Jan 11 11:55:39 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: 2022-01-11T10:55:39.599Z 29462 TID-n26 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-29462", :url=>"redis://localhost:6379/0"}
Jan 11 11:55:42 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:55:46 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:55:46 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
Jan 11 11:55:53 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: User with login admin already exists, not seeding as admin.
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: 390 extra bytes after the deserialized object
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `block in find_execution_plans'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `map'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence_adapters/sequel.rb:74:in `find_execution_plans'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/persistence.rb:60:in `find_execution_plans'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/app/jobs/application_job.rb:11:in `spawn_if_missing'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `each'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:43:in `block in initialize!'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `tap'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/rails.rb:38:in `initialize!'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
Jan 11 11:55:57 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29462]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILURE
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.
Jan 11 11:55:57 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service failed.
Jan 11 11:55:58 katello01.infra.aza.nu systemd[1]: dynflow-sidekiq@orchestrator.service holdoff time over, scheduling restart.
Jan 11 11:55:58 katello01.infra.aza.nu systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
Jan 11 11:55:58 katello01.infra.aza.nu systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jan 11 11:56:00 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: 2022-01-11T10:56:00.110Z 29539 TID-n3r INFO: GitLab reliable fetch activated!
Jan 11 11:56:00 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: 2022-01-11T10:56:00.111Z 29539 TID-n2j INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-29539", :url=>"redis://localhost:6379/0"}
Jan 11 11:56:03 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Jan 11 11:56:06 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0/lib/foreman_puppet/register.rb:155: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLES
Jan 11 11:56:06 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[29539]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was here
2

Hey @cazzen2k4
Sorry to hear you’re having issues.
Did you get any further?

3

Hi @mcorr

Unfortunately not. I’ve taken a snapshot of the server so I can revert to before the upgrade. I’ve tried the upgrade multiple times but without any success.

This log-output sticks out to me looking at the dynflow-sidekiq@orchestrator log above.

an 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
Jan 11 11:33:23 katello01.infra.aza.nu dynflow-sidekiq@orchestrator[16243]: DETAIL:  Key (typname, typnamespace)=(dynflow_output_chunks_id_seq, 2200) already exists.
4

I’ll try and get some eyes on this !

5

Can anyone from @katello advise as to what the next steps to troubleshoot this might be?

6

Please try clearing out ALL your tasks first:

1 Like
7

Hi @lfu,

Clearing all the tasks worked for us, the installation succeeded and things seem to be working as expected. Thank you!

However we are still experiencing an error message when trying to run ‘foreman-rake db:seed --trace’. I’m not sure if this has any impact on the current installation or if it might prohibit us from making a successful upgrade of foreman in the future. See output below.

Output

[root@katello01 ~]# foreman-rake db:seed --trace
** Invoke db:seed (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
** Execute db:load_config
** Execute db:seed
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke db:load_config 
** Execute db:abort_if_pending_migrations
** Invoke dynflow:abort_if_pending_migrations (first_time)
** Invoke environment 
** Execute dynflow:abort_if_pending_migrations
User with login admin already exists, not seeding as admin.
rake aborted!
ActiveRecord::RecordNotDestroyed: Failed to destroy the record
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:951:in `_raise_record_not_destroyed'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:551:in `destroy!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/has_many_association.rb:99:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/has_many_association.rb:99:in `delete_records'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:385:in `remove_records'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:376:in `block in delete_or_destroy'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:135:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:134:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:376:in `delete_or_destroy'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:198:in `destroy'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/autosave_association.rb:393:in `block in save_collection_association'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/autosave_association.rb:393:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/autosave_association.rb:393:in `save_collection_association'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/autosave_association.rb:186:in `block in add_autosave_association_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/autosave_association.rb:158:in `instance_eval'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/autosave_association.rb:158:in `block in define_non_cyclic_method'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:428:in `block in make_lambda'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:238:in `block in halting_and_conditional'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:517:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:517:in `invoke_after'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:825:in `_run_update_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:335:in `_update_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/timestamp.rb:123:in `_update_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:905:in `create_or_update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:327:in `block in create_or_update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:135:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:327:in `create_or_update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/timestamp.rb:128:in `create_or_update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:503:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:53:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `block in save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:48:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/models/job_template.rb:68:in `import_raw!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/db/seeds.d/75-job_templates.rb:10:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/db/seeds.d/75-job_templates.rb:6:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/db/seeds.d/75-job_templates.rb:6:in `block (2 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/db/seeds.d/75-job_templates.rb:3:in `block in <top (required)>'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/db/seeds.d/75-job_templates.rb:2:in `<top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `block in load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `load'
/usr/share/foreman/app/services/foreman_seeder.rb:51:in `block (3 levels) in execute'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/usr/share/foreman/app/services/foreman_seeder.rb:50:in `block (2 levels) in execute'
/usr/share/foreman/app/services/foreman_seeder.rb:44:in `each'
/usr/share/foreman/app/services/foreman_seeder.rb:44:in `block in execute'
/usr/share/foreman/lib/foreman/advisory_lock_manager.rb:20:in `block in with_transaction_lock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
/usr/share/foreman/lib/foreman/advisory_lock_manager.rb:17:in `with_transaction_lock'
/usr/share/foreman/app/services/foreman_seeder.rb:38:in `execute'
/usr/share/foreman/db/seeds.rb:14:in `<top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `block in load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `load'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in `block in load'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in `load'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:559:in `load_seed'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/tasks/database_tasks.rb:440:in `load_seed'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/databases.rake:331:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby27/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby27/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => db:seed
1 Like
8

Hi @lfu and @mcorr,

I don’t see above issue anymore after upgrading to foreman 3.1.1. Foreman-rake db:seed is no longer complaining. Thx guys!

1 Like