Upgrade Failed with Errors

Kreggenwinkel · September 16, 2021, 7:52am

Problem:
We are triying to upgrade our Demo Foreman, but are failing with errors.
Previous Version [Foreman 2.5 and Katello 4.1]
Upgraded Version [Foreman 3.0 and Katello 4.2]

Expected outcome:
Upgrade complete with no problems

Foreman and Proxy versions:
Previous Version [Foreman 2.5 and Katello 4.1]
Upgraded Version [Foreman 3.0 and Katello 4.2]

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:

We get the following Outputs, when we run foreman-installer
Processing: foreman_install_messages…
Processing: foreman_install_messages.log…

And we got a previous error while trying to update the packages

--> Finished Dependency Resolution
Error: Package: tfm-rubygem-qpid_proton-0.34.0-3.el7.x86_64 (@katello)
           Requires: qpid-proton-c = 0.34.0
           Removing: qpid-proton-c-0.34.0-2.el7.x86_64 (@epel)
               qpid-proton-c = 0.34.0-2.el7
           Updated By: qpid-proton-c-0.35.0-1.el7.x86_64 (epel)
               qpid-proton-c = 0.35.0-1.el7
           Available: qpid-proton-c-0.14.0-2.el7.x86_64 (extras)
               qpid-proton-c = 0.14.0-2.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Then we run the following command to exclude this package
yum -x qpid-proton-c,python36-qpid-proton update

We run those rake commands bevor the update:

 foreman-rake db:seed
 foreman-rake apipie:cache
 foreman-rake katello:upgrade_check

Kreggenwinkel · September 16, 2021, 7:54am

First Half of output

2021-09-16 09:33:22 [NOTICE] [root] Loading installer configuration. This will take some time.
2021-09-16 09:33:26 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2021-09-16 09:33:26 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Executing: katello-certs-check -c "/etc/ssl/certs/foreman_server.foo.bar.crt" -k "/etc/ssl/certs/foreman_server.foo.bar.key" -b "/etc/ssl/certs/chain256.pem"
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Checking server certificate encoding: [OK]

Checking expiration of certificate: [OK]

Checking expiration of CA bundle: [OK]

Checking if server certificate has CA:TRUE flag [OK]

Checking for private key passphrase: [OK]

Checking to see if the private key matches the certificate: [OK]

Checking CA bundle against the certificate file: [OK]

Checking CA bundle size: 2
[OK]

Checking Subject Alt Name on certificate [OK]

Checking if any Subject Alt Name on certificate matches the Subject CN
[OK]

Checking Key Usage extension on certificate for Key Encipherment [OK]

Checking for use of shortname as CN
[OK]

Validation succeeded


To install the Katello server with the custom certificates, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/etc/pki/tls/certs/foreman_server.foo.bar.crt" \
                      --certs-server-key "/etc/pki/tls/certs/foreman_server.foo.bar.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/chain256.pem"

To update the certificates on a currently running Katello installation, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/etc/pki/tls/certs/foreman_server.foo.bar.crt" \
                      --certs-server-key "/etc/pki/tls/certs/foreman_server.foo.bar.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/chain256.pem" \
                      --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy
2021-09-16 09:33:37 [NOTICE] [configure] Starting system configuration.
2021-09-16 09:33:50 [NOTICE] [configure] 250 configuration steps out of 1929 steps complete.
2021-09-16 09:34:01 [NOTICE] [configure] 500 configuration steps out of 1933 steps complete.
2021-09-16 09:34:02 [NOTICE] [configure] 750 configuration steps out of 1935 steps complete.
2021-09-16 09:34:10 [NOTICE] [configure] 1000 configuration steps out of 1939 steps complete.
2021-09-16 09:34:33 [NOTICE] [configure] 1250 configuration steps out of 1944 steps complete.
2021-09-16 09:34:35 [NOTICE] [configure] 1500 configuration steps out of 1945 steps complete.
2021-09-16 09:36:04 [ERROR ] [configure] Systemd start for foreman failed!
2021-09-16 09:36:04 [ERROR ] [configure] journalctl log for foreman:
2021-09-16 09:36:04 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:36:04 CEST. --
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar systemd[1]: Stopping Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: [1004] === puma shutdown: 2021-09-16 09:31:49 +0200 ===
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: [1004] - Goodbye!
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: [1004] - Gracefully shutting down workers...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: Exiting
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar systemd[1]: Stopped Foreman.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar systemd[1]: Starting Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:54 foreman_server.foo.bar foreman[28718]: => Booting Puma
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:54 foreman_server.foo.bar foreman[28718]: => Rails 6.0.3.7 application starting in production
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:54 foreman_server.foo.bar foreman[28718]: => Run `rails server --help` for more startup options
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:58 foreman_server.foo.bar foreman[28718]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Enabling systemd notification integration
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] Puma starting in cluster mode...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Puma version: 5.3.2 (ruby 2.7.3-p183) ("Sweetnighter")
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *  Min threads: 0
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *  Max threads: 16
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *  Environment: production
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *   Master PID: 28718
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *      Workers: 2
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *     Restarts: (✔) hot (✖) phased
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Preloading application
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Activated unix:///run/foreman.sock
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] Use Ctrl-C to stop
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Starting control server on unix:///usr/share/foreman/tmp/sockets/pumactl.sock
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:21 foreman_server.foo.bar foreman[28718]: [28718] - Worker 0 (PID: 28742) booted in 0.13s, phase: 0
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:21 foreman_server.foo.bar foreman[28718]: [28718] - Worker 1 (PID: 28747) booted in 0.14s, phase: 0
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:21 foreman_server.foo.bar systemd[1]: Started Foreman.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar systemd[1]: Stopping Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar foreman[28718]: [28718] === puma shutdown: 2021-09-16 09:33:06 +0200 ===
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar foreman[28718]: [28718] - Goodbye!
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar foreman[28718]: [28718] - Gracefully shutting down workers...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:07 foreman_server.foo.bar foreman[28718]: Exiting
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:07 foreman_server.foo.bar systemd[1]: Stopped Foreman.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:35:55 foreman_server.foo.bar systemd[1]: Starting Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:00 foreman_server.foo.bar foreman[32042]: => Booting Puma
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:00 foreman_server.foo.bar foreman[32042]: => Rails 6.0.3.7 application starting in production
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:00 foreman_server.foo.bar foreman[32042]: => Run `rails server --help` for more startup options
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:02 foreman_server.foo.bar foreman[32042]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: Exiting
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: /usr/share/foreman/app/models/role.rb:335:in `permission_records': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations] (Foreman::PermissionMissingException)
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
2021-09-16 09:36:04 [ERROR ] [configure] /Stage[main]/Foreman::Service/Service[foreman]/ensure: change from 'stopped' to 'running' failed: Systemd start for foreman failed!
2021-09-16 09:36:04 [ERROR ] [configure] journalctl log for foreman:
2021-09-16 09:36:04 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:36:04 CEST. --
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar systemd[1]: Stopping Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: [1004] === puma shutdown: 2021-09-16 09:31:49 +0200 ===
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: [1004] - Goodbye!
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: [1004] - Gracefully shutting down workers...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar foreman[1004]: Exiting
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar systemd[1]: Stopped Foreman.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:49 foreman_server.foo.bar systemd[1]: Starting Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:54 foreman_server.foo.bar foreman[28718]: => Booting Puma
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:54 foreman_server.foo.bar foreman[28718]: => Rails 6.0.3.7 application starting in production
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:54 foreman_server.foo.bar foreman[28718]: => Run `rails server --help` for more startup options
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:31:58 foreman_server.foo.bar foreman[28718]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Enabling systemd notification integration
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] Puma starting in cluster mode...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Puma version: 5.3.2 (ruby 2.7.3-p183) ("Sweetnighter")
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *  Min threads: 0
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *  Max threads: 16
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *  Environment: production
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *   Master PID: 28718
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *      Workers: 2
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] *     Restarts: (✔) hot (✖) phased
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Preloading application
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Activated unix:///run/foreman.sock
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] Use Ctrl-C to stop
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:20 foreman_server.foo.bar foreman[28718]: [28718] * Starting control server on unix:///usr/share/foreman/tmp/sockets/pumactl.sock
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:21 foreman_server.foo.bar foreman[28718]: [28718] - Worker 0 (PID: 28742) booted in 0.13s, phase: 0
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:21 foreman_server.foo.bar foreman[28718]: [28718] - Worker 1 (PID: 28747) booted in 0.14s, phase: 0
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:32:21 foreman_server.foo.bar systemd[1]: Started Foreman.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar systemd[1]: Stopping Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar foreman[28718]: [28718] === puma shutdown: 2021-09-16 09:33:06 +0200 ===
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar foreman[28718]: [28718] - Goodbye!
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:06 foreman_server.foo.bar foreman[28718]: [28718] - Gracefully shutting down workers...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:07 foreman_server.foo.bar foreman[28718]: Exiting
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:33:07 foreman_server.foo.bar systemd[1]: Stopped Foreman.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:35:55 foreman_server.foo.bar systemd[1]: Starting Foreman...
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:00 foreman_server.foo.bar foreman[32042]: => Booting Puma
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:00 foreman_server.foo.bar foreman[32042]: => Rails 6.0.3.7 application starting in production
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:00 foreman_server.foo.bar foreman[32042]: => Run `rails server --help` for more startup options
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:02 foreman_server.foo.bar foreman[32042]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: Exiting
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: /usr/share/foreman/app/models/role.rb:335:in `permission_records': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations] (Foreman::PermissionMissingException)
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:36:04 [ERROR ] [configure] Sep 16 09:36:04 foreman_server.foo.bar foreman[32042]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'


2021-09-16 09:41:22 [ERROR ] [configure] Systemd start for dynflow-sidekiq@orchestrator failed!
2021-09-16 09:41:22 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2021-09-16 09:41:22 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:41:22 CEST. --
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:13 foreman_server.foo.bar systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:14 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:14.880Z 33050 TID-qku INFO: GitLab reliable fetch activated!
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:14 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:14.881Z 33050 TID-qlu INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-33050", :url=>"redis://localhost:6379/0"}
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:14 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:14.888Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:15 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:15.910Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:16 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:16.917Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:17 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:17.919Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:18 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:18.923Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:19 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:19.947Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:20 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:20.968Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:21 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:21.982Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations]
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:335:in `permission_records'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:368:in `block (2 levels) in role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:367:in `block in role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'

Kreggenwinkel · September 16, 2021, 7:54am

second half of the output:

2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:77:in `block in <top (required)>'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `instance_eval'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `register'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:3:in `<top (required)>'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/engine.rb:37:in `block in <class:Engine>'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:347:in `each'
2021-09-16 09:41:22 [ERROR ] [configure] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[orchestrator]/Service[dynflow-sidekiq@orchestrator]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@orchestrator failed!
2021-09-16 09:41:22 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2021-09-16 09:41:22 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:41:22 CEST. --
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:13 foreman_server.foo.bar systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:14 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:14.880Z 33050 TID-qku INFO: GitLab reliable fetch activated!
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:14 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:14.881Z 33050 TID-qlu INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-33050", :url=>"redis://localhost:6379/0"}
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:14 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:14.888Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:15 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:15.910Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:16 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:16.917Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:17 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:17.919Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:18 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:18.923Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:19 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:19.947Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:20 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:20.968Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:21 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: 2021-09-16T07:41:21.982Z 33050 TID-qlu ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations]
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:335:in `permission_records'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:368:in `block (2 levels) in role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:367:in `block in role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:366:in `role'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:77:in `block in <top (required)>'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `instance_eval'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `register'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:3:in `<top (required)>'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/engine.rb:37:in `block in <class:Engine>'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
2021-09-16 09:41:22 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar dynflow-sidekiq@orchestrator[33050]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:347:in `each'
2021-09-16 09:41:33 [ERROR ] [configure] Systemd start for dynflow-sidekiq@worker-1 failed!
2021-09-16 09:41:33 [ERROR ] [configure] journalctl log for dynflow-sidekiq@worker-1:
2021-09-16 09:41:33 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:41:33 CEST. --
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar systemd[1]: Starting Foreman jobs daemon - worker-1 on sidekiq...
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:24 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:24.217Z 33132 TID-qns INFO: GitLab reliable fetch activated!
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:24 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:24.217Z 33132 TID-qmc INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-33132", :url=>"redis://localhost:6379/0"}
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:24 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:24.226Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:25 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:25.249Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:26 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:26.255Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:27 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:27.259Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:28 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:28.277Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:29 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:29.341Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:30 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:30.343Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:31 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:31.360Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:32.365Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations]
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:335:in `permission_records'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:368:in `block (2 levels) in role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:367:in `block in role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:366:in `role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:77:in `block in <top (required)>'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `instance_eval'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `register'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:3:in `<top (required)>'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/engine.rb:37:in `block in <class:Engine>'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
2021-09-16 09:41:33 [ERROR ] [configure] /Stage[main]/Foreman::Service/Foreman::Dynflow::Pool[worker]/Foreman::Dynflow::Worker[worker-1]/Service[dynflow-sidekiq@worker-1]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@worker-1 failed!
2021-09-16 09:41:33 [ERROR ] [configure] journalctl log for dynflow-sidekiq@worker-1:
2021-09-16 09:41:33 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:41:33 CEST. --
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:22 foreman_server.foo.bar systemd[1]: Starting Foreman jobs daemon - worker-1 on sidekiq...
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:24 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:24.217Z 33132 TID-qns INFO: GitLab reliable fetch activated!
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:24 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:24.217Z 33132 TID-qmc INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-33132", :url=>"redis://localhost:6379/0"}
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:24 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:24.226Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:25 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:25.249Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:26 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:26.255Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:27 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:27.259Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:28 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:28.277Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:29 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:29.341Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:30 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:30.343Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:31 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:31.360Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: 2021-09-16T07:41:32.365Z 33132 TID-qmc ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations]
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:335:in `permission_records'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:368:in `block (2 levels) in role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:367:in `block in role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:366:in `role'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:77:in `block in <top (required)>'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `instance_eval'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `register'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:3:in `<top (required)>'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/engine.rb:37:in `block in <class:Engine>'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
2021-09-16 09:41:33 [ERROR ] [configure] Sep 16 09:41:32 foreman_server.foo.bar dynflow-sidekiq@worker-1[33132]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
2021-09-16 09:41:44 [ERROR ] [configure] Systemd start for dynflow-sidekiq@worker-hosts-queue-1 failed!
2021-09-16 09:41:44 [ERROR ] [configure] journalctl log for dynflow-sidekiq@worker-hosts-queue-1:
2021-09-16 09:41:44 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:41:44 CEST. --
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:33 foreman_server.foo.bar systemd[1]: Starting Foreman jobs daemon - worker-hosts-queue-1 on sidekiq...
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:34 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:34.911Z 33240 TID-qj0 INFO: GitLab reliable fetch activated!
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:34 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:34.912Z 33240 TID-qk0 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-33240", :url=>"redis://localhost:6379/0"}
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:34 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:34.917Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:35 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:35.922Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:36 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:36.925Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:37 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:37.932Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:38 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:38.935Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:39 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:39.948Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:40 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:40.985Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:42 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:42.006Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:43 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:43.028Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:44.033Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations]
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:335:in `permission_records'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:368:in `block (2 levels) in role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:367:in `block in role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:366:in `role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:77:in `block in <top (required)>'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `instance_eval'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `register'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:3:in `<top (required)>'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/engine.rb:37:in `block in <class:Engine>'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
2021-09-16 09:41:44 [ERROR ] [configure] /Stage[main]/Katello::Application/Foreman::Dynflow::Pool[worker-hosts-queue]/Foreman::Dynflow::Worker[worker-hosts-queue-1]/Service[dynflow-sidekiq@worker-hosts-queue-1]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@worker-hosts-queue-1 failed!
2021-09-16 09:41:44 [ERROR ] [configure] journalctl log for dynflow-sidekiq@worker-hosts-queue-1:
2021-09-16 09:41:44 [ERROR ] [configure] -- Logs begin at Wed 2021-09-08 16:18:28 CEST, end at Thu 2021-09-16 09:41:44 CEST. --
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:33 foreman_server.foo.bar systemd[1]: Starting Foreman jobs daemon - worker-hosts-queue-1 on sidekiq...
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:34 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:34.911Z 33240 TID-qj0 INFO: GitLab reliable fetch activated!
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:34 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:34.912Z 33240 TID-qk0 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-33240", :url=>"redis://localhost:6379/0"}
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:34 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:34.917Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:35 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:35.922Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:36 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:36.925Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:37 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:37.932Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:38 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:38.935Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:39 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:39.948Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:40 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:40.985Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:42 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:42.006Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:43 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:43.028Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: 2021-09-16T07:41:44.033Z 33240 TID-qk0 ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_smart_proxies", "view_job_templates", "create_job_invocations", "create_template_invocations", :view_smart_proxies, :view_job_templates, :create_job_invocations, :create_template_invocations]
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:335:in `permission_records'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:171:in `find_for_permission_removal'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:212:in `remove_permissions!'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/role.rb:86:in `ignore_locking'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:52:in `update_plugin_role_permissions'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:23:in `process_role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:368:in `block (2 levels) in role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:367:in `block in role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:366:in `role'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:77:in `block in <top (required)>'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `instance_eval'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/app/registries/foreman/plugin.rb:106:in `register'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/register.rb:3:in `<top (required)>'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-6.4.1/lib/foreman_ansible/engine.rb:37:in `block in <class:Engine>'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-09-16 09:41:44 [ERROR ] [configure] Sep 16 09:41:44 foreman_server.foo.bar dynflow-sidekiq@worker-hosts-queue-1[33240]: /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
2021-09-16 09:41:44 [NOTICE] [configure] 1750 configuration steps out of 1945 steps complete.
2021-09-16 09:42:40 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

Dirk · September 16, 2021, 7:58am

Unfortunatly you are hitting a known issue which is already worked on.