Foreman RPM nightly pipeline failed:
https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/1224/
foreman-pipeline-foreman-nightly-centos8-stream-install (passed) (remote job)
foreman-pipeline-foreman-nightly-centos7-install (passed) (remote job)
foreman-pipeline-foreman-nightly-centos8-install (passed) (remote job)
foreman-pipeline-foreman-nightly-centos7-upgrade (failed) (remote job)
foreman-pipeline-foreman-nightly-centos8-upgrade (failed) (remote job)
Looks like upgrades are failing on a dynflow issue now:
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] /Service[dynflow-sidekiq@orchestrator]: Systemd restart for dynflow-sidekiq@orchestrator failed!e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] -- Logs begin at Sat 2021-10-16 21:57:13 UTC, end at Sat 2021-10-16 22:36:00 UTC. --e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: [368B blob data]e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/rh/rh-ruby27/root/usr/share/ruby/json/common.rb:156:in `parse'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:14:in `load'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:21:in `load'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:122:in `load'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:362:in `load_data'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `block in find_coordinator_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `map'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `find_coordinator_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator.rb:366:in `find_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:80:in `heartbeat'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: [ concurrent-ruby ]e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:113:in `on_message'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:32:14 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[28887]: [ concurrent-ruby ]e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:34 pipe-up-foreman-nightly-centos7.n29.example.com systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:36 pipe-up-foreman-nightly-centos7.n29.example.com systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:36 pipe-up-foreman-nightly-centos7.n29.example.com systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:39 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: 2021-10-16T22:35:39.205Z 32280 TID-o0c INFO: GitLab reliable fetch activated!e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:39 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: 2021-10-16T22:35:39.207Z 32280 TID-o1c INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-32280", :url=>"redis://localhost:6379/0"}e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:45 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:46 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-2.0.0.alpha.2/lib/foreman_puppet/register.rb:120: warning: already initialized constant Foreman::Plugin::RbacSupport::AUTO_EXTENDED_ROLESe[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:35:46 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:5: warning: previous definition of AUTO_EXTENDED_ROLES was heree[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: 286 extra bytes after the deserialized objecte[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/persistence_adapters/sequel.rb:260:in `block in find_coordinator_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/persistence_adapters/sequel.rb:260:in `map'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/coordinator.rb:366:in `find_records'e[0m
[2021-10-16T22:36:50.018Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/coordinator.rb:372:in `find_worlds'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/world/invalidation.rb:131:in `worlds_validity_check'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/world/invalidation.rb:118:in `perform_validity_checks'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/rails.rb:46:in `block in initialize!'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/rails.rb:38:in `tap'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.1/lib/dynflow/rails.rb:38:in `initialize!'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com dynflow-sidekiq@orchestrator[32280]: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com systemd[1]: dynflow-sidekiq@orchestrator.service: main process exited, code=exited, status=1/FAILUREe[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.e[0m
[2021-10-16T22:36:50.019Z] e[0;31m 2021-10-16 22:36:00 [ERROR ] [configure] Oct 16 22:36:00 pipe-up-foreman-nightly-centos7.n29.example.com systemd[1]: Unit dynflow-sidekiq@orchestrator.service entered failed state.e[0mThis topic was automatically closed 7 days after the last reply. New replies are no longer allowed.