Foreman/Katello 2.0 -> 2.1 / 3.15 -> 3.16 unsuccessful on numerous attempts

Problem:
Foreman UI does not load and I continue to receive an error upgrading from 2.0 to 2.1 with Katello 3.15 to 3.16.

I took a snapshot of the VM prior to the upgrade and can roll back anytime but would really like to try and get this server operational ASAP if at all possible. Thank you.

Expected outcome:
Completed and successful upgrade.

Foreman and Proxy versions:
2.0 -> 2.1 (current and broken)

Foreman and Proxy plugin versions:
foreman-2.1.4-1.el7.noarch
foreman-cli-2.1.4-1.el7.noarch
foreman-debug-2.1.4-1.el7.noarch
foreman-dynflow-sidekiq-2.1.4-1.el7.noarch
foreman-ec2-2.1.4-1.el7.noarch
foreman.example.com-apache-1.0-35.noarch
foreman.example.com-foreman-client-1.0-35.noarch
foreman.example.com-foreman-proxy-1.0-35.noarch
foreman.example.com-foreman-proxy-client-1.0-35.noarch
foreman.example.com-puppet-client-1.0-35.noarch
foreman.example.com-qpid-broker-1.0-35.noarch
foreman.example.com-qpid-client-cert-1.0-35.noarch
foreman.example.com-qpid-router-client-1.0-35.noarch
foreman.example.com-qpid-router-server-1.0-35.noarch
foreman.example.com-tomcat-1.0-29.noarch
foreman-installer-2.1.4-1.el7.noarch
foreman-installer-katello-2.1.4-1.el7.noarch
foreman-openstack-2.1.4-1.el7.noarch
foreman-postgresql-2.1.4-1.el7.noarch
foreman-proxy-2.1.4-1.el7.noarch
foreman-proxy-content-3.16.2-1.el7.noarch
foreman-release-2.1.4-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-2.1.4-1.el7.noarch
foreman-service-2.1.4-1.el7.noarch
foreman-vmware-2.1.4-1.el7.noarch
katello-3.16.2-1.el7.noarch
katello-agent-3.3.5-4.el7.noarch
katello-certs-tools-2.7.0-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.16.2-1.el7.noarch
katello-debug-3.16.2-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.3.5-4.el7.noarch
katello-host-tools-fact-plugin-3.3.5-4.el7.noarch
katello-repos-3.16.2-2.el7.noarch
katello-selinux-3.4.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.14.1-1.el7.noarch

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:

[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LoadError: cannot load such file -- apipie/middleware/checksum_in_headers
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/config/application.rb:5:in `<top (required)>'
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/Rakefile:1:in `<top (required)>'
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)
[ERROR 2021-01-12T14:11:54 verbose]  '/usr/sbin/foreman-env ; /usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:183:in `sync'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:241:in `sync'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:136:in `sync_if_needed'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:90:in `block in perform_changes'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `each'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `perform_changes'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:148:in `as_logging_destination'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:193:in `block (2 levels) in apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `block in apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:191:in `apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:423:in `run_internal'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:251:in `block in run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:218:in `run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2021-01-12T14:11:54 verbose] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-env ; /usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Dependency Exec[foreman-rake-db:migrate] has failures: true
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Skipping because of failed dependencies
[ INFO 2021-01-12T14:11:54 verbose]  Class[Apache::Service]: Unscheduling all events on Class[Apache::Service]
[ WARN 2021-01-12T14:11:54 verbose]  /Service[httpd]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Katello::Application/Foreman_config_entry[pulp_client_cert]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Katello::Application/Foreman_config_entry[pulp_client_key]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_enabled]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_address]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_ssl_ca_file]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_ssl_certificate]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_ssl_private_key]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman::Plugin::Puppetdb/Foreman_config_entry[puppetdb_api_version]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Stage[main]/Foreman/Foreman::Rake[apipie:cache:index]/Exec[foreman-rake-apipie:cache:index]: Skipping because of failed dependencies
[ WARN 2021-01-12T14:11:54 verbose]  /Service[dynflow-sidekiq@worker-hosts-queue]: Skipping because of failed dependencies

Here is the output of running db:migrate manually:

$ /usr/sbin/foreman-rake db:migrate
Rubocop not loaded.
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
== 20191017151119 AddExternalIpamIdToSubnets: migrating =======================
-- add_column(:subnets, :externalipam_id, :integer)
   -> 0.0078s
-- add_column(:subnets, :externalipam_group, :text)
   -> 0.0010s
-- add_index(:subnets, :externalipam_id)
   -> 0.0175s
== 20191017151119 AddExternalIpamIdToSubnets: migrated (0.0269s) ==============

== 20200204034123 AddPulp3HrefToRepositoryErratum: migrating ==================
-- add_column(:katello_repository_errata, :erratum_pulp3_href, :string)
   -> 0.0084s
-- add_index(:katello_repository_errata, :erratum_pulp3_href)
   -> 0.0591s
== 20200204034123 AddPulp3HrefToRepositoryErratum: migrated (0.0679s) =========

== 20200213184848 CreateEvrType: migrating ====================================
-- enable_extension("evr")
   -> 0.0329s
-- add_column(:katello_rpms, :evr, :evr_t)
   -> 0.0026s
-- add_column(:katello_installed_packages, :evr, :evr_t)
   -> 0.0021s
-- create_trigger(:evr_insert_trigger_katello_rpms, {:on=>:katello_rpms})
   -> 0.0052s
-- create_trigger(:evr_update_trigger_katello_rpms, {:on=>:katello_rpms})
   -> 0.0018s
-- create_trigger(:evr_insert_trigger_katello_installed_packages, {:on=>:katello_installed_packages})
   -> 0.0017s
-- create_trigger(:evr_update_trigger_katello_installed_packages, {:on=>:katello_installed_packages})
   -> 0.0018s
-- execute("        update katello_rpms SET evr = (ROW(coalesce(epoch::numeric,0),\n                                           rpmver_array(coalesce(version,'empty'))::evr_array_item[],\n                                           rpmver_array(coalesce(release,'empty'))::evr_array_item[])::evr_t);\n\n        update katello_installed_packages SET evr = (ROW(coalesce(epoch::numeric,0),\n                                                         rpmver_array(coalesce(version,'empty'))::evr_array_item[],\n                                                         rpmver_array(coalesce(release,'empty'))::evr_array_item[])::evr_t);\n")
   -> 32.6454s
-- add_index(:katello_rpms, [:name, :arch, :evr])
   -> 2.4815s
-- add_index(:katello_erratum_packages, [:erratum_id, :nvrea])
   -> 0.5832s
== 20200213184848 CreateEvrType: migrated (35.7597s) ==========================

== 20200217110708 AlterSessionSequenceToCycle: migrating ======================
-- change_column(:sessions, :id, :bigint)
   -> 0.0813s
== 20200217110708 AlterSessionSequenceToCycle: migrated (0.0883s) =============

== 20200304201026 CreateKatelloModuleStreamRpms: migrating ====================
-- create_table(:katello_module_stream_rpms, {})
   -> 0.0189s
-- add_index(:katello_module_stream_rpms, [:module_stream_id, :rpm_id], {:unique=>true, :name=>:katello_msrpm_module_stream_rpms_uniq})
   -> 0.0044s
-- add_foreign_key("katello_module_stream_rpms", "katello_module_streams", {:column=>"module_stream_id", :name=>:katello_msrpm_mod_stream_id_fk})
   -> 0.0079s
-- add_foreign_key("katello_module_stream_rpms", "katello_rpms", {:column=>"rpm_id", :name=>:katello_msrpm_rpm_id_fk})
   -> 0.0048s
== 20200304201026 CreateKatelloModuleStreamRpms: migrated (0.0366s) ===========

== 20200311131527 DropSmartVariablePermissions: migrating =====================
== 20200311131527 DropSmartVariablePermissions: migrated (0.1948s) ============

== 20200311134414 ChangeMatchToTextInLookupvalue: migrating ===================
-- change_column(:lookup_values, :match, :text, {:limit=>nil})
   -> 0.0094s
== 20200311134414 ChangeMatchToTextInLookupvalue: migrated (0.0096s) ==========

== 20200318135632 RenameReportTemplates: migrating ============================
== 20200318135632 RenameReportTemplates: migrated (0.1306s) ===================

== 20200324210540 RenameDefaultVariableLookupPath: migrating ==================
== 20200324210540 RenameDefaultVariableLookupPath: migrated (0.0105s) =========

== 20200326164755 ChangeOidcAudienceSettingType: migrating ====================
== 20200326164755 ChangeOidcAudienceSettingType: migrated (0.0102s) ===========

== 20200402190048 AddHiddenValueToTemplateInput: migrating ====================
-- add_column(:template_inputs, :hidden_value, :boolean, {:default=>false, :null=>false})
   -> 0.0033s
== 20200402190048 AddHiddenValueToTemplateInput: migrated (0.0034s) ===========

== 20200406113832 DropVariableLookupKey: migrating ============================
== 20200406113832 DropVariableLookupKey: migrated (0.0835s) ===================

== 20200407171926 CreateContentGuard: migrating ===============================
-- create_table(:katello_content_guards, {})
   -> 0.0082s
-- add_column(:katello_distribution_references, :content_guard_href, :string)
   -> 0.0011s
== 20200407171926 CreateContentGuard: migrated (0.0096s) ======================

== 20200421201839 UpdateAnsibleInvTemplateName: migrating =====================
== 20200421201839 UpdateAnsibleInvTemplateName: migrated (0.0033s) ============

== 20200427151946 DropRackspaceCr: migrating ==================================
== 20200427151946 DropRackspaceCr: migrated (0.0844s) =========================

== 20200506163345 AddApplicabilityIndicestoKatelloHostAvailableModuleStreams: migrating
-- add_index(:katello_host_available_module_streams, [:host_id, :available_module_stream_id, :status], {:name=>"rpm_and_module_applicability_related_indices"})
   -> 0.0236s
== 20200506163345 AddApplicabilityIndicestoKatelloHostAvailableModuleStreams: migrated (0.0239s)

== 20200511204005 CreateKatelloHostQueueElements: migrating ===================
-- create_table(:katello_host_queue_elements)
   -> 0.0146s
== 20200511204005 CreateKatelloHostQueueElements: migrated (0.0148s) ==========

== 20200513092446 ChangeConfigChartClass: migrating ===========================
== 20200513092446 ChangeConfigChartClass: migrated (0.1349s) ==================

== 20200517215015 RenameBookmarksController: migrating ========================
== 20200517215015 RenameBookmarksController: migrated (0.0205s) ===============

== 20200519093217 DropDynflowAllowDangerousActionsSetting: migrating ==========
== 20200519093217 DropDynflowAllowDangerousActionsSetting: migrated (0.0021s) =

== 20200525073201 DropPuppetClassIdFromLookUpKey: migrating ===================
-- remove_column(:lookup_keys, :puppetclass_id, :integer)
   -> 0.0040s
== 20200525073201 DropPuppetClassIdFromLookUpKey: migrated (0.0042s) ==========

== 20200526200422 EncryptRootRepositoryUpstreamPassword: migrating ============
-- change_column(:katello_root_repositories, :upstream_password, :text)
   -> 0.0024s
== 20200526200422 EncryptRootRepositoryUpstreamPassword: migrated (0.8871s) ===

== 20200709021250 AddOriginalModulesToContentViewModuleStreamFilter: migrating
-- add_column(:katello_content_view_filters, :original_module_streams, :boolean, {:default=>false, :null=>false})
   -> 0.0030s
== 20200709021250 AddOriginalModulesToContentViewModuleStreamFilter: migrated (0.0032s)

== 20200709160133 SetEmptyFilterTaxonomySearchNil: migrating ==================
== 20200709160133 SetEmptyFilterTaxonomySearchNil: migrated (0.0522s) =========

== 20200721142707 RemoveDuplicateKatelloPoolsIndex: migrating =================
-- remove_index(:katello_pools, {:name=>"index_pools_on_cp_id"})
   -> 0.0075s
== 20200721142707 RemoveDuplicateKatelloPoolsIndex: migrated (0.0077s) ========

== 20200803065041 MigratePortOverridesForAnsible: migrating ===================
== 20200803065041 MigratePortOverridesForAnsible: migrated (0.0440s) ==========

Not sure if this is important to mention, but this error comes up a few times shortly after the upgrade error occurred.

2021-01-12T13:25:58 [E|kat|81262a20] Katello::Errors::ConnectionRefusedException: A backend service [ Candlepin ] is unreachable
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/katello/http_resource.rb:95:in `rescue in get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/katello/http_resource.rb:84:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/katello/resources/candlepin/consumer.rb:87:in `serials'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:263:in `serials'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:35:in `repackage_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/middleware/event_daemon.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:25:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-01-12T13:25:58 [I|app|81262a20] Completed 500 Internal Server Error in 51ms (Views: 0.4ms | ActiveRecord: 27.0ms)

Here is the output of
/usr/sbin/foreman-rake -T

Rubocop not loaded.
rake about                                                                           # List versions of all Rails frameworks and the environment
rake action_mailbox:ingress:exim                                                     # Relay an inbound email from Exim to Action Mailbox (URL and INGRESS_PASSWORD required)
rake action_mailbox:ingress:postfix                                                  # Relay an inbound email from Postfix to Action Mailbox (URL and INGRESS_PASSWORD required)
rake action_mailbox:ingress:qmail                                                    # Relay an inbound email from Qmail to Action Mailbox (URL and INGRESS_PASSWORD required)
rake action_mailbox:install                                                          # Copy over the migration
rake action_text:install                                                             # Copy over the migration, stylesheet, and JavaScript files
rake active_storage:install                                                          # Copy over the migration needed to the application
rake apipie:cache                                                                    # Generate cache to avoid production dependencies on markup languages
rake apipie:cache:index                                                              # Generate cache index for all languages (override with FOREMAN_APIPIE_LANGS environment ...
rake apipie:client                                                                   # Generate CLI client for API documented with apipie gem
rake apipie:convert_examples                                                         # Convert your examples from the old yaml into the new json format
rake apipie:did_swagger_change[version,swagger_content_type_input,filename_suffix]   # Did swagger output change since the last execution of this task?
rake apipie:static[version]                                                          # Generate static documentation
rake apipie:static_json[version]                                                     # Generate static documentation json
rake apipie:static_swagger_json[version,swagger_content_type_input,filename_suffix]  # Generate static swagger json
rake apipie:update_from_routes                                                       # Update api description in controllers base on routes
rake apipie_dsl:cache[include_json]                                                  # Generate cache to avoid production dependencies on markup languages
rake apipie_dsl:static[version,include_json]                                         # Generate static documentation
rake apipie_dsl:static_json                                                          # Generate static documentation json
rake app:template                                                                    # Applies the template supplied by LOCATION=(/path/to/template) or URL
rake app:update                                                                      # Update configs and some other initially generated files (or use just update:configs or ...
rake assets:clean[keep]                                                              # Remove old compiled assets
rake assets:clobber                                                                  # Remove compiled assets
rake assets:environment                                                              # Load asset compile environment
rake assets:precompile                                                               # Compile all the assets named in config.assets.precompile
rake audits:expire                                                                   # Expire or anonymize old audits automatically
rake auth_source_external:create                                                     # Creates an external authentication source named "External", if no external authenticati...
rake bootdisk:generate:full_host                                                     # Generate a full boot disk for a specific host with the OS bootloader included
rake bootdisk:generate:generic                                                       # Generate a generic boot disk
rake bootdisk:generate:host                                                          # Generate a static boot disk for a specific host
rake bootdisk:generate:subnet                                                        # Generate a subnet disk for a specific subnet
rake bundler:deps                                                                    # Show dependencies in nicely-formatted output
rake cache_digests:dependencies                                                      # Lookup first-level dependencies for TEMPLATE (like messages/show or comments/_comment.h...
rake cache_digests:nested_dependencies                                               # Lookup nested dependencies for TEMPLATE (like messages/show or comments/_comment.html)
rake clean                                                                           # Remove any temporary products
rake clobber                                                                         # Remove any generated files
rake config                                                                          # Configure Foreman in-database settings, see rake -- config --help for more details
rake db:auth_sources_ldap:decrypt                                                    # Decrypt LDAP authentication source fields
rake db:auth_sources_ldap:encrypt                                                    # Encrypt LDAP authentication source fields
rake db:compute_resources:decrypt                                                    # Decrypt compute resource fields
rake db:compute_resources:encrypt                                                    # Encrypt compute resource fields
rake db:convert:prod2dev                                                             # Convert/import production data to development
rake db:create                                                                       # Creates the database from DATABASE_URL or config/database.yml for the current RAILS_ENV...
rake db:decrypt_all                                                                  # Decrypt all passwords (compute resources, LDAP authentication sources) using the encryp...
rake db:drop                                                                         # Drops the database from DATABASE_URL or config/database.yml for the current RAILS_ENV (...
rake db:dump                                                                         # Make a dump of your database
rake db:encrypt_all                                                                  # Encrypt all passwords (compute resources, LDAP authentication sources) using
rake db:environment:set                                                              # Set the environment value for the database
rake db:fixtures:load                                                                # Loads fixtures into the current environment's database
rake db:import_dump                                                                  # Import a database dump
rake db:migrate                                                                      # Migrate the database (options: VERSION=x, VERBOSE=false, SCOPE=blog)
rake db:migrate:status                                                               # Display status of migrations
rake db:prepare                                                                      # Runs setup if database does not exist, or runs migrations if it does
rake db:rollback                                                                     # Rolls the schema back to the previous version (specify steps w/ STEP=n)
rake db:schema:cache:clear                                                           # Clears a db/schema_cache.yml file
rake db:schema:cache:dump                                                            # Creates a db/schema_cache.yml file
rake db:schema:dump                                                                  # Creates a db/schema.rb file that is portable against any DB supported by Active Record
rake db:schema:load                                                                  # Loads a schema.rb file into the database
rake db:seed                                                                         # Loads the seed data from db/seeds.rb
rake db:seed:replant                                                                 # Truncates tables of each database for current environment and loads the seeds
rake db:sequence:reset                                                               # Reset PostgreSQL sequences to their max values
rake db:sessions:clear                                                               # Clear the sessions table
rake db:sessions:create                                                              # Creates a sessions migration for use with ActiveRecord::SessionStore
rake db:sessions:trim                                                                # Trim old sessions from the table (default: > 30 days)
rake db:setup                                                                        # Creates the database, loads the schema, and initializes with the seed data (use db:rese...
rake db:structure:dump                                                               # Dumps the database structure to db/structure.sql
rake db:structure:load                                                               # Recreates the databases from the structure.sql file
rake db:version                                                                      # Retrieves the current schema version number
rake deface:clean                                                                    # Removes all precompiled override templates
rake deface:failures_by_virtual_path[virtual_path]                                   # Report on failing overrides for a partial/template
rake deface:get_result[virtual_path]                                                 # Get the resulting markup for a partial/template
rake deface:precompile                                                               # Precompiles overrides into template files
rake deface:test_all                                                                 # Load and apply all overrides, and output results
rake deface:test_selector[virtual_path,selector]                                     # Applies selectors to given partial/template, and returns match(s) source
rake dynflow:client                                                                  # Sets up the environment to act as a Dynflow client
rake dynflow:executor                                                                # In development mode, the Dynflow executor is part of the web server process
rake exception:codes                                                                 # List all error codes
rake exports:csv                                                                     # Database exporting
rake facts:clean                                                                     # Removes facts without any values
rake fix_db_cache                                                                    # Fix user groups and authorization cache by removing all cached records and recreating them
rake foreman_ansible:install:migrations                                              # Copy migrations from foreman_ansible to application
rake foreman_bootdisk:install:migrations                                             # Copy migrations from foreman_bootdisk to application
rake foreman_discovery:install:migrations                                            # Copy migrations from foreman_discovery to application
rake foreman_openscap:bulk_upload:directory[directory]                               # Bulk upload SCAP content from directory
rake foreman_openscap:clean_messages                                                 # Clean duplicate messages for ArfReport
rake foreman_openscap:clean_reports_without_proxy                                    # Delete ArfReports without OpenSCAP proxy
rake foreman_openscap:install:migrations                                             # Copy migrations from foreman_openscap to application
rake foreman_remote_execution:example:task                                           # Example Task
rake foreman_remote_execution:install:migrations                                     # Copy migrations from foreman_remote_execution to application
rake foreman_rescue:install:migrations                                               # Copy migrations from foreman_rescue to application
rake foreman_tasks:cleanup:config                                                    # Show the current configuration for auto-cleanup
rake foreman_tasks:cleanup:run                                                       # Clean tasks based on filter and age
rake foreman_tasks:export_tasks                                                      # Export dynflow tasks based on filter
rake foreman_tasks:generate_task_actions                                             # Generate missing values for action column in foreman_tasks_tasks table
rake foreman_tasks:install:migrations                                                # Copy migrations from foreman_tasks to application
rake foreman_virt_who_configure:cleanup                                              # Clean default data created by this plugin, this will permanently delete the data
rake foreman_virt_who_configure:install:migrations                                   # Copy migrations from foreman_virt_who_configure to application
rake foreman_wreckingball:install:migrations                                         # Copy migrations from foreman_wreckingball to application
rake foreman_wreckingball:vmware:sync                                                # Synchonize VMware compute resource data
rake hosts:scan_out_of_sync                                                          # This task search hosts that have been out of sync for longer than
rake interfaces:clean                                                                # Removes old interfaces that match ignored interfaces pattern setting
rake jenkins:foreman_snapshot_management                                             # Test ForemanSnapshotManagement with XML output for jenkins
rake katello:change_download_policy                                                  # Change the download policy of all repos
rake katello:check_ping                                                              # Runs a katello ping and prints out the statuses of each service
rake katello:clean_backend_objects                                                   # Cleans backend objects (hosts) that are missing in one or more backend systems
rake katello:clean_old_file_repos                                                    # Cleans up file repos that were moved to /pulp/isos/<org name>/
rake katello:clean_published_repo_directories                                        # Cleans up obsolete directories and estimates time to republish current directories
rake katello:correct_puppet_environments                                             # Correct missing pulp repositories for puppet environments
rake katello:correct_repositories                                                    # Correct missing pulp repositories
rake katello:delete_orphaned_content                                                 # Remove orphaned and unneeded content/repos from a smart proxy
rake katello:install:migrations                                                      # Copy migrations from katello to application
rake katello:publish_unpublished_repositories                                        # Check for repositories that have not been published since their last sync, and republis...
rake katello:pulp3_content_switchover                                                # Runs a Pulp 3 migration of pulp3 hrefs to pulp ids for supported content types
rake katello:pulp3_migration                                                         # Runs a Pulp 2 to 3 Content Migration for supported types
rake katello:pulp3_post_migration_check                                              # Runs a post Pulp3 migration check for supported content types
rake katello:receptor:extract_orgs                                                   # Extract Organization details for Receptor and write to OUTPUT_FILE
rake katello:refresh_pulp_repo_details                                               # Refresh repository metadata for all repositories
rake katello:refresh_sync_schedule                                                   # Refresh pulp sync schedules
rake katello:regenerate_repo_metadata                                                # Regnerate metadata for all repositories
rake katello:regenerate_ueber_certs[organization]                                    # Regenerates the ueber cert for each organization in candlepin, can be passed an organiz...
rake katello:reimport                                                                # Reimports information from backend systems
rake katello:reset_backends                                                          # Resets the Foreman/Katello development environment
rake katello:rubocop                                                                 # Runs Rubocop style checker on Katello code
rake katello:rubocop:jenkins                                                         # Runs Rubocop style checker with xml output for Jenkins
rake katello:sync_ansible_job_templates                                              # Synchronize Ansible job templates with lates community templates
rake katello:unify_hosts                                                             # Unify hosts that have registered with both fqdn and shortname
rake katello:update_default_http_proxy                                               # Sets the content default HTTP proxy to an existing HTTP proxy based on supplied URL
rake katello:upgrades:3.10:clear_invalid_repo_credentials                            # Clear invalid credentials for repositories
rake katello:upgrades:3.10:update_gpg_key_urls                                       # Update repositories with API V1 GPG URLs
rake katello:upgrades:3.11:import_yum_metadata                                       # Import yum metadata file units from pulp
rake katello:upgrades:3.11:update_puppet_repos                                       # update puppet repos to regenerate pulp configuration
rake katello:upgrades:3.12:remove_pulp2_notifier                                     # removes the unused pulp2 notifier
rake katello:upgrades:3.13:republish_deb_metadata[input_file]                        # Regenerate the metadata for repositories affected by pulp_deb migration 0004
rake katello:upgrades:3.15:reindex_rpm_modular[input_file]                           # Reindex the is_modular attribute of an rpm
rake katello:upgrades:3.15:set_sub_facet_dmi_uuid[input_file]                        # Set the DMI UUID on Host::SubscriptionFacet from facts
rake katello:upgrades:3.16:update_applicable_el8_hosts[input_file]                   # Update the applicability calculations for Rhel8 hosts
rake katello:upgrades:3.8:clear_checksum_type                                        # Clear checksum type for on-demand repositories
rake katello:virt_who_report                                                         # Report on hypervisors without VDC subscriptions and guests consuming virtual entitlements
rake locale:find                                                                     # Extract strings from model and from codebase
rake locale:find_code                                                                # Extract strings from codebase
rake locale:find_model                                                               # Extract strings from model
rake locale:pack                                                                     # Alias for gettext:pack
rake locale:po_to_json                                                               # Alias for gettext:po_to_json
rake log                                                                             # Log output of ActiveRecord actions to stdout
rake log:clear                                                                       # Truncates all/specified *.log files in log/ to zero bytes (specify which logs with LOGS...
rake middleware                                                                      # Prints out your Rack middleware stack
rake models:consolidate[dryrun]                                                      # Reduce the many vendor supplied model names to a few sensible model definitions
rake notifications:clean[time,group,blueprint]                                       # Clean expired notification
rake orchestration:dhcp:add_missing                                                  # Orchestration maintainance tasks
rake parameters:cast_key_types_and_values                                            # Auto detect the key type & update formatted value in parameters
rake parameters:reset_priorities                                                     # Reset parameter priorities in case they were changed
rake permissions:reset                                                               # Create or reset "admin" user permissions to defaults
rake pkg:deb                                                                         # Create DEB package with `debuild`
rake pkg:generate_source                                                             # Generate package source tar.bz2, supply ref=<tag> for tags
rake plugin:apipie:cache[engine]                                                     # Generate apipie:cache for plugin - called via rake plugin:apipie:cache[plugin_name]
rake plugin:assets:precompile[plugin]                                                # Compile plugin assets - called via rake plugin:assets:precompile[plugin_name]
rake plugin:list                                                                     # List Installed plugins
rake plugin:validate_roles                                                           # Validate permissions for built-in roles
rake puppet:import:external_nodes                                                    # Import your hosts classes and parameters classifications from another external node source
rake puppet:import:hosts_and_facts[dir]                                              # Imports hosts and facts from existings YAML files, use dir= to override default directory
rake puppetdb_foreman:install:migrations                                             # Copy migrations from puppetdb_foreman to application
rake reports:daily                                                                   # Send an email notifications such as summarising hosts Puppet reports (and lack of it), ...
rake reports:expire                                                                  # Expire Reports automatically
rake rescan_reports_origins                                                          # Rescan existing reports without origin and tries to find correct origin
rake restart                                                                         # Restart app by touching tmp/restart.txt
rake rss:create_notifications                                                        # Create notifications from an RSS feed
rake secret                                                                          # Generate a cryptographically secure secret key (this is typically used to generate a se...
rake secure_headers:generate_hashes                                                  # Generate config/secure_headers_generated_hashes.yml
rake security:generate_encryption_key                                                # Generate new encryption key
rake security:generate_token[path]                                                   # Generate new security token
rake seed:forgeries                                                                  # Generate various forgeries
rake snapshots:generate                                                              # Generate snapshots
rake stats                                                                           # Report code statistics (KLOCs, etc) from the application or engine
rake telemetry:metrics                                                               # List all metrics
rake telemetry:prometheus_statsd[output]                                             # Generate exporter mapping for statsd_exporter
rake templates:render                                                                # Render templates from a directory
rake test                                                                            # Runs all tests in test folder except system ones
rake test:api                                                                        # Run tests for api
rake test:db                                                                         # Run tests quickly, but also reset db
rake test:discovery                                                                  # Run tests for discovery
rake test:foreman_bootdisk                                                           # Run tests for foreman_bootdisk
rake test:foreman_openscap                                                           # Run tests for foreman_openscap
rake test:foreman_remote_execution                                                   # Run tests for {:foreman_remote_execution=>["db:test:prepare"]}
rake test:foreman_rescue                                                             # Run tests for foreman_rescue
rake test:foreman_snapshot_management                                                # Run tests for foreman_snapshot_management
rake test:foreman_virt_who_configure                                                 # Run tests for foreman_virt_who_configure
rake test:foreman_wreckingball                                                       # Run tests for foreman_wreckingball
rake test:graphql                                                                    # Run tests for graphql
rake test:katello                                                                    # Run the entire Katello plugin test suite
rake test:katello:spec                                                               # Run the Katello plugin spec test suite
rake test:katello:spec:routing                                                       # Run the Katello plugin spec test suite
rake test:katello:test                                                               # Run the Katello plugin unit test suite
rake test:katello:test:actions                                                       # Run the Katello plugin unit actions test suite
rake test:katello:test:controllers                                                   # Run the Katello plugin unit controllers test suite
rake test:katello:test:glue                                                          # Run the Katello plugin unit glue test suite
rake test:katello:test:lib                                                           # Run the Katello plugin unit lib test suite
rake test:katello:test:live_scenarios                                                # Delete scenario vcr cassettes and run the katello scenario tests in live mode
rake test:katello:test:models                                                        # Run the Katello plugin unit models test suite
rake test:katello:test:services                                                      # Run the Katello plugin unit glue test suite
rake test:puppetdb_foreman                                                           # Run tests for puppetdb_foreman
rake test:system                                                                     # Run system tests only
rake time:zones[country_or_offset]                                                   # List all time zones, list by two-letter country code (`rails time:zones[US]`), or list ...
rake tmp:clear                                                                       # Clear cache, socket and screenshot files from tmp/ (narrow w/ tmp:cache:clear, tmp:sock...
rake tmp:create                                                                      # Creates tmp directories for cache, sockets, and pids
rake trends:counter                                                                  # Create Trend counts
rake trends:reduce                                                                   # Reduces amount of points for each trend group
rake upgrade:run                                                                     # This task runs predefined upgrade steps
rake webpack:compile                                                                 # Compile webpack bundles: overriding the rake task from webpack-rails to be
rake yarn:install                                                                    # Install all JavaScript dependencies as specified via Yarn
rake zeitwerk:check                                                                  # Checks project structure for Zeitwerk compatibility

Come to find out, foreman-maintain service status was showing all services in good [OK] status but the foreman service was disabled and stopped?

Problem #1 - foreman-maintain service status not showing the current state of all services.
Problem #2 - Why was the foreman service disabled and stopped in the first place?

The installer still fails with the error in the verbose output in the OP.

I await any feedback.

Thanks