Dynflow failed to start on upgrade from foreman 3.2.1 to 3.3

Problem: Dynflow failed to start after upgrade from Foreman 3.2.1 to 3.3

Expected outcome: Expected outcome is dynflow will start normally

Foreman and Proxy versions: Foreman 3.3

Foreman and Proxy plugin versions: Foreman 3.3

Distribution and version: RHEL 8.6

foreman-installer
2022-06-30 23:31:08 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-06-30 23:31:11 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-06-30 23:31:11 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-06-30 23:31:14 [NOTICE] [configure] Starting system configuration.
2022-06-30 23:31:25 [NOTICE] [configure] 250 configuration steps out of 1215 steps complete.
2022-06-30 23:31:25 [NOTICE] [configure] 500 configuration steps out of 1220 steps complete.
2022-06-30 23:31:25 [NOTICE] [configure] 750 configuration steps out of 1224 steps complete.
2022-06-30 23:31:42 [NOTICE] [configure] 1000 configuration steps out of 1224 steps complete.
2022-06-30 23:31:46 [ERROR ] [configure] Systemd start for dynflow-sidekiq@orchestrator failed!
2022-06-30 23:31:46 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2022-06-30 23:31:46 [ERROR ] [configure] -- Logs begin at Mon 2022-06-27 19:54:55 PDT, end at Thu 2022-06-30 23:31:46 PDT. --
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: undefined method `error' for Dynflow::DelayedPlan:Class
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:81:in `rescue in new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:72:in `new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/persistence.rb:117:in `load_delayed_plan'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/app/jobs/application_job.rb:13:in `block in spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/app/jobs/application_job.rb:12:in `select'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/app/jobs/application_job.rb:12:in `spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `each'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `tap'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:292:in `boot_system'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:46:in `run'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/gems/gems/sidekiq-5.2.10/bin/sidekiq:12:in `<top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/bin/sidekiq:23:in `load'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/bin/sidekiq:23:in `<main>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: dynflow-sidekiq@orchestrator.service: Service RestartSec=1s expired, scheduling restart.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: dynflow-sidekiq@orchestrator.service: Scheduled restart job, restart counter is at 120.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  dynflow-sidekiq@orchestrator[290989]: Ignoring msgpack-1.5.2 because its extensions are not built. Try: gem pristine msgpack --version 1.5.2
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:53  dynflow-sidekiq@orchestrator[290989]: 2022-07-01T06:26:53.482Z 290989 TID-69tl INFO: GitLab reliable fetch activated!
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:53  dynflow-sidekiq@orchestrator[290989]: 2022-07-01T06:26:53.483Z 290989 TID-69s5 INFO: Booting Sidekiq 5.2.10 with redis options {:id=>"Sidekiq-server-PID-290989", :url=>"redis://localhost:6379/0"}
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: undefined method `error' for Dynflow::DelayedPlan:Class
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:81:in `rescue in new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:72:in `new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/persistence.rb:117:in `load_delayed_plan'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/app/jobs/application_job.rb:13:in `block in spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/app/jobs/application_job.rb:12:in `select'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/app/jobs/application_job.rb:12:in `spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `each'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `tap'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:292:in `boot_system'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:46:in `run'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/gems/gems/sidekiq-5.2.10/bin/sidekiq:12:in `<top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[orchestrator]/Service[dynflow-sidekiq@orchestrator]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@orchestrator failed!
2022-06-30 23:31:46 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2022-06-30 23:31:46 [ERROR ] [configure] -- Logs begin at Mon 2022-06-27 19:54:55 PDT, end at Thu 2022-06-30 23:31:46 PDT. --
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: undefined method `error' for Dynflow::DelayedPlan:Class
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:81:in `rescue in new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:72:in `new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/persistence.rb:117:in `load_delayed_plan'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/app/jobs/application_job.rb:13:in `block in spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/app/jobs/application_job.rb:12:in `select'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/app/jobs/application_job.rb:12:in `spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `each'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `tap'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:292:in `boot_system'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:46:in `run'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/share/gems/gems/sidekiq-5.2.10/bin/sidekiq:12:in `<top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/bin/sidekiq:23:in `load'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  dynflow-sidekiq@orchestrator[290965]: /usr/bin/sidekiq:23:in `<main>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:50  systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: dynflow-sidekiq@orchestrator.service: Service RestartSec=1s expired, scheduling restart.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: dynflow-sidekiq@orchestrator.service: Scheduled restart job, restart counter is at 120.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:51  dynflow-sidekiq@orchestrator[290989]: Ignoring msgpack-1.5.2 because its extensions are not built. Try: gem pristine msgpack --version 1.5.2
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:53  dynflow-sidekiq@orchestrator[290989]: 2022-07-01T06:26:53.482Z 290989 TID-69tl INFO: GitLab reliable fetch activated!
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:26:53  dynflow-sidekiq@orchestrator[290989]: 2022-07-01T06:26:53.483Z 290989 TID-69s5 INFO: Booting Sidekiq 5.2.10 with redis options {:id=>"Sidekiq-server-PID-290989", :url=>"redis://localhost:6379/0"}
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: undefined method `error' for Dynflow::DelayedPlan:Class
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:81:in `rescue in new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/delayed_plan.rb:72:in `new_from_hash'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/persistence.rb:117:in `load_delayed_plan'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/app/jobs/application_job.rb:13:in `block in spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/app/jobs/application_job.rb:12:in `select'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/app/jobs/application_job.rb:12:in `spawn_if_missing'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `each'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:43:in `block in initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `tap'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/local/share/gems/gems/dynflow-1.6.7/lib/dynflow/rails.rb:38:in `initialize!'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:292:in `boot_system'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/cli.rb:46:in `run'
2022-06-30 23:31:46 [ERROR ] [configure] Jun 30 23:27:05  dynflow-sidekiq@orchestrator[290989]: /usr/share/gems/gems/sidekiq-5.2.10/bin/sidekiq:12:in `<top (required)>'
2022-06-30 23:31:48 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/foreman.log

Team, In addition to the above I am observing an error as listed attached when trying to access Hosts > All hosts.

This can be marked as resolved, I end up rebuilding the entire foreman/katello integration in a new server. Too many issues resolting from the upgrade to foreman 3.3 and katelo 4.5. After clean install things appear to be stable, at least for now.