Foreman Katello installation error systemd start fordynflow-sidekiq@worker-hosts-queue-1 failed

Problem:
2021-07-22 16:21:10 [NOTICE] [configure] Starting system configuration.
2021-07-22 16:21:29 [NOTICE] [configure] 250 configuration steps out of 1904 steps complete.
2021-07-22 16:21:42 [NOTICE] [configure] 500 configuration steps out of 1905 steps complete.
2021-07-22 16:21:44 [NOTICE] [configure] 750 configuration steps out of 1908 steps complete.
2021-07-22 16:21:48 [NOTICE] [configure] 1000 configuration steps out of 1912 steps complete.
2021-07-22 16:21:49 [NOTICE] [configure] 1250 configuration steps out of 1916 steps complete.
2021-07-22 16:24:21 [ERROR ] [configure] Systemd start for dynflow-sidekiq@worker-hosts-queue-1 failed!
2021-07-22 16:24:21 [ERROR ] [configure] journalctl log for dynflow-sidekiq@worker-hosts-queue-1:

Expected outcome:
Be able to install Foreman and Katello

Foreman and Proxy versions:
2.5 though not yet installed so na

Foreman and Proxy plugin versions:

Distribution and version:

Linux vx1916.uprr.com 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Other relevant data:

foreman-installer --scenario katello

Could you get us more details? Output of journalctl -u dynflow-sidekiq@worker-hosts-queue-1 would be nice to have

Apparently it keeps trying and is filling up the /var/log filesystem but here are the errors:
Jul 23 05:53:03 vx1916.uprr.com systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service: main process exited, code=exited, status=1/FAILURE
Jul 23 05:53:03 vx1916.uprr.com systemd[1]: Failed to start Foreman jobs daemon - worker-hosts-queue-1 on sidekiq.
Jul 23 05:53:03 vx1916.uprr.com systemd[1]: Unit dynflow-sidekiq@worker-hosts-queue-1.service entered failed state.
Jul 23 05:53:03 vx1916.uprr.com systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service failed.
Jul 23 05:53:04 vx1916.uprr.com systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service holdoff time over, scheduling restart.

here is a few lines before that:
Jul 23 05:53:03 vx1916.uprr.com dynflow-sidekiq@worker-hosts-queue-1[16107]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in boot_system' Jul 23 05:53:03 vx1916.uprr.com dynflow-sidekiq@worker-hosts-queue-1[16107]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in run’
Jul 23 05:53:03 vx1916.uprr.com dynflow-sidekiq@worker-hosts-queue-1[16107]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in <top (required)>' Jul 23 05:53:03 vx1916.uprr.com dynflow-sidekiq@worker-hosts-queue-1[16107]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in load’
Jul 23 05:53:03 vx1916.uprr.com dynflow-sidekiq@worker-hosts-queue-1[16107]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `’

Please print complete logs and use code blocks.

the complete log file is over 50k in lines
[root[DEV]@vx1916 log]# journalctl -u dynflow-sidekiq@worker-hosts-queue-1 | wc -l
51896

You really need to paste your logs into a code block. Otherwise it’s pretty unreadable…

Put three backquotes on a seperate line before and after:

```
logs
```

looks like this:

logs
Jul 23 09:51:39 vx1916 systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service: main process exited, code=exited, status=1/FAILURE
Jul 23 09:51:39 vx1916 systemd[1]: Failed to start Foreman jobs daemon - worker-hosts-queue-1 on sidekiq.
Jul 23 09:51:39 vx1916 systemd[1]: Unit dynflow-sidekiq@worker-hosts-queue-1.service entered failed state.
Jul 23 09:51:39 vx1916 systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service failed.
Jul 23 09:51:40 vx1916 systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service holdoff time over, scheduling restart.
Jul 23 09:51:40 vx1916 systemd[1]: Stopped Foreman jobs daemon - worker-hosts-queue-1 on sidekiq.
Jul 23 09:51:40 vx1916 systemd[1]: Starting Foreman jobs daemon - worker-hosts-queue-1 on sidekiq...
Jul 23 09:51:46 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: 2021-07-23T14:51:46.073Z 11476 TID-8jk INFO: GitLab reliable fetch activated!
Jul 23 09:51:46 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: 2021-07-23T14:51:46.092Z 11476 TID-8l8 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-11476", :url=>"redis://localhost:6379/0"}
Jul 23 09:51:54 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/graphql-1.8.14/lib/graphql/schema/mutation.rb:70: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Jul 23 09:51:54 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/graphql-1.8.14/lib/graphql/schema/member/has_fields.rb:51: warning: The called method `field' is defined here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/version.rb:4: warning: already initialized constant JSON::VERSION
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/version.rb:4: warning: previous definition of VERSION was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/version.rb:5: warning: already initialized constant JSON::VERSION_ARRAY
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/version.rb:5: warning: previous definition of VERSION_ARRAY was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/version.rb:6: warning: already initialized constant JSON::VERSION_MAJOR
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/version.rb:6: warning: previous definition of VERSION_MAJOR was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/version.rb:7: warning: already initialized constant JSON::VERSION_MINOR
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/version.rb:7: warning: previous definition of VERSION_MINOR was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/version.rb:8: warning: already initialized constant JSON::VERSION_BUILD
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/version.rb:8: warning: previous definition of VERSION_BUILD was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/common.rb:100: warning: already initialized constant JSON::NaN
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/common.rb:100: warning: previous definition of NaN was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/common.rb:102: warning: already initialized constant JSON::Infinity
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/common.rb:102: warning: previous definition of Infinity was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/common.rb:104: warning: already initialized constant JSON::MinusInfinity
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/common.rb:104: warning: previous definition of MinusInfinity was here
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/gems/gems/json-2.3.0/lib/json/common.rb:129: warning: already initialized constant JSON::UnparserError
Jul 23 09:51:57 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/common.rb:129: warning: previous definition of UnparserError was here
Jul 23 09:51:59 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
Jul 23 09:51:59 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Jul 23 09:52:01 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/will_paginate-3.1.7/lib/will_paginate/deprecation.rb:35: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
Jul 23 09:52:01 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/will_paginate-3.1.7/lib/will_paginate/deprecation.rb:35: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
Jul 23 09:52:02 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/definition.rb:309: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Jul 23 09:52:02 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/definition.rb:27: warning: The called method `initialize' is defined here
Jul 23 09:52:06 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/models/concerns/has_many_common.rb:48: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Jul 23 09:52:06 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1370: warning: The called method `has_many' is defined here
Jul 23 09:52:06 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/models/concerns/has_many_common.rb:53: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Jul 23 09:52:06 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1826: warning: The called method `has_and_belongs_to_many' is defined here
Jul 23 09:52:06 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/models/concerns/has_many_common.rb:77: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Jul 23 09:52:06 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1657: warning: The called method `belongs_to' is defined here
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: undefined method `id' for nil:NilClass
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/models/foreman_tasks/task.rb:64:in `block in <class:Task>'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:184:in `map_value'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:253:in `preprocess_parameters'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:228:in `block in sql_test'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:228:in `each'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:228:in `sql_test'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:562:in `sql_test'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:509:in `to_single_field_sql'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:521:in `to_sql'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:54:in `build_find_params'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_builder.rb:22:in `build_query'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/definition.rb:328:in `block in register_named_scope!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/models/filter.rb:14:in `validate'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:428:in `block in make_lambda'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:605:in `block (2 levels) in default_terminator'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:604:in `catch'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:604:in `block in default_terminator'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:201:in `block in halting'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `block in invoke_before'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `each'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `invoke_before'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:134:in `run_callbacks'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:825:in `_run_validate_callbacks'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.7/lib/active_model/validations.rb:406:in `run_validations!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.7/lib/active_model/validations/callbacks.rb:117:in `block in run_validations!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:135:in `run_callbacks'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:825:in `_run_validation_callbacks'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.7/lib/active_model/validations/callbacks.rb:117:in `run_validations!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.7/lib/active_model/validations.rb:337:in `valid?'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:68:in `valid?'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:84:in `perform_validations'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:53:in `save!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `block in save!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `save!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:48:in `save!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/services/cache_manager.rb:14:in `map'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/services/cache_manager.rb:14:in `block in create_new_filter_cache'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/services/cache_manager.rb:13:in `create_new_filter_cache'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/app/services/cache_manager.rb:22:in `recache!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/config/initializers/fix_cache.rb:10:in `<top (required)>'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `load'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `block in load'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `load'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in `block in load'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in `load'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:666:in `block in load_config_initializer'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:182:in `instrument'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:665:in `load_config_initializer'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:625:in `block (2 levels) in <class:Engine>'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:624:in `each'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:624:in `block in <class:Engine>'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:60:in `run_initializers'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/application.rb:363:in `initialize!'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:92:in `require'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:92:in `require'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /usr/share/foreman/extras/dynflow-sidekiq.rb:16:in `<top (required)>'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:92:in `require'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:92:in `require'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
Jul 23 09:52:12 vx1916 dynflow-sidekiq@worker-hosts-queue-1[11476]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
Jul 23 09:52:12 vx1916 systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service: main process exited, code=exited, status=1/FAILURE
Jul 23 09:52:12 vx1916 systemd[1]: Failed to start Foreman jobs daemon - worker-hosts-queue-1 on sidekiq.
Jul 23 09:52:12 vx1916 systemd[1]: Unit dynflow-sidekiq@worker-hosts-queue-1.service entered failed state.
Jul 23 09:52:12 vx1916 systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service failed.
Jul 23 09:52:13 vx1916 systemd[1]: dynflow-sidekiq@worker-hosts-queue-1.service holdoff time over, scheduling restart.
Jul 23 09:52:13 vx1916 systemd[1]: Stopped Foreman jobs daemon - worker-hosts-queue-1 on sidekiq.
Jul 23 09:52:13 vx1916 systemd[1]: Starting Foreman jobs daemon - worker-hosts-queue-1 on sidekiq...

There is the exact error, which comes from here:

It looks like it assumes a user is logged in because the current_user filter is specified, but there isn’t a user logged in. It doesn’t handle that.

I’m not sure where this comes from. @aruzicka?

Could this have come from this option? --foreman-initial-admin-username foreman

Apparently foreman tries to fix permission and filter cache on startup. If a user does not exist at this point, something must have gone wrong with seeding.

That would be surprising. What other options did you pass to the installer?

Here is the full line less the password:
foreman-installer --scenario katello --enable-foreman-plugin-ansible --enable-foreman-plugin-bootdisk --enable-foreman-plugin-discovery --enable-foreman-plugin-hooks --enable-foreman-plugin-openscap --enable-foreman-plugin-remote-execution --enable-foreman-plugin-templates --foreman-initial-organization “Union Pacific Railroad” --foreman-initial-location “UPC Omaha Nebraska” --foreman-initial-admin-username foreman --foreman-initial-admin-password

Thank you everyone who looked at this. We discovered the issue, we had an error within our yum repo that pointed to the GPG key for puppet in that the path was incorrect. Not sure why it did not error during the first time we installed katello, but this time it generated an error for the key allowing us to correct the issue and perform a successful installation.