Problem:
Foreman UI does not load and I continue to receive an error upgrading from 2.0 to 2.1 with Katello 3.15 to 3.16.
I took a snapshot of the VM prior to the upgrade and can roll back anytime but would really like to try and get this server operational ASAP if at all possible. Thank you.
Expected outcome:
Completed and successful upgrade.
Foreman and Proxy versions:
2.0 -> 2.1 (current and broken)
Foreman and Proxy plugin versions:
foreman-2.1.4-1.el7.noarch
foreman-cli-2.1.4-1.el7.noarch
foreman-debug-2.1.4-1.el7.noarch
foreman-dynflow-sidekiq-2.1.4-1.el7.noarch
foreman-ec2-2.1.4-1.el7.noarch
foreman.example.com-apache-1.0-35.noarch
foreman.example.com-foreman-client-1.0-35.noarch
foreman.example.com-foreman-proxy-1.0-35.noarch
foreman.example.com-foreman-proxy-client-1.0-35.noarch
foreman.example.com-puppet-client-1.0-35.noarch
foreman.example.com-qpid-broker-1.0-35.noarch
foreman.example.com-qpid-client-cert-1.0-35.noarch
foreman.example.com-qpid-router-client-1.0-35.noarch
foreman.example.com-qpid-router-server-1.0-35.noarch
foreman.example.com-tomcat-1.0-29.noarch
foreman-installer-2.1.4-1.el7.noarch
foreman-installer-katello-2.1.4-1.el7.noarch
foreman-openstack-2.1.4-1.el7.noarch
foreman-postgresql-2.1.4-1.el7.noarch
foreman-proxy-2.1.4-1.el7.noarch
foreman-proxy-content-3.16.2-1.el7.noarch
foreman-release-2.1.4-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-2.1.4-1.el7.noarch
foreman-service-2.1.4-1.el7.noarch
foreman-vmware-2.1.4-1.el7.noarch
katello-3.16.2-1.el7.noarch
katello-agent-3.3.5-4.el7.noarch
katello-certs-tools-2.7.0-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.16.2-1.el7.noarch
katello-debug-3.16.2-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.3.5-4.el7.noarch
katello-host-tools-fact-plugin-3.3.5-4.el7.noarch
katello-repos-3.16.2-2.el7.noarch
katello-selinux-3.4.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.14.1-1.el7.noarch
Distribution and version:
CentOS Linux release 7.9.2009 (Core)
Other relevant data:
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LoadError: cannot load such file -- apipie/middleware/checksum_in_headers
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/config/application.rb:5:in `<top (required)>'
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/Rakefile:1:in `<top (required)>'
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)
[ERROR 2021-01-12T14:11:54 verbose] '/usr/sbin/foreman-env ; /usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:183:in `sync'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:241:in `sync'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:136:in `sync_if_needed'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:90:in `block in perform_changes'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `each'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `perform_changes'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:148:in `as_logging_destination'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:193:in `block (2 levels) in apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `block in apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:191:in `apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:423:in `run_internal'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:251:in `block in run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:218:in `run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-env ; /usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Dependency Exec[foreman-rake-db:migrate] has failures: true
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Skipping because of failed dependencies
[ INFO 2021-01-12T14:11:54 verbose] Class[Apache::Service]: Unscheduling all events on Class[Apache::Service]
[ WARN 2021-01-12T14:11:54 verbose] /Service[httpd]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Katello::Application/Foreman_config_entry[pulp_client_cert]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Katello::Application/Foreman_config_entry[pulp_client_key]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_enabled]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_address]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_ssl_ca_file]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_ssl_certificate]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_ssl_private_key]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_api_version]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Stage[main]/Foreman/Foreman::Rake[apipie:cache:index]/Exec[foreman-rake-apipie:cache:index]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose] /Service[dynflow-sidekiq@worker-hosts-queue]: Skipping because of failed dependencies