[ContentMigration] Migrate Foreman 2.3.2 + Katello 3.18 to newer fails, "Validation failed: resources missing"

Problem:
I am attempting to upgrade from Foreman 2.3.2 and in the process I need to migrate from Pulp 2 to 3. When I try to run foreman-maintain content migration-stats or foreman-maintain content prepare it gives no output. When I try to then run foreman-maintain content switchover it gives me the following error output:

[root@foreman foreman]# foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 enabled
Migration failed, You will want to investigate: https://foreman.localdomain/foreman_tasks/tasks/0ff8df19-0530-4587-81ef-86ef830fba5e
rake aborted!
ForemanTasks::TaskError: Task 0ff8df19-0530-4587-81ef-86ef830fba5e: Katello::Errors::Pulp3Error: Validation failed: resources missing {'repositories': ['7636016e-ea37-46e6-8353-39f1b54cff7d', '17ebf2a9-7a98-48fd-bcde-bca2744e2368'], 'repositories_missing_importers': ['7636016e-ea37-46e6-8353-39f1b54cff7d', '17ebf2a9-7a98-48fd-bcde-bca2744e2368'], 'repositories_missing_distributors': ['7636016e-ea37-46e6-8353-39f1b54cff7d', '17ebf2a9-7a98-48fd-bcde-bca2744e2368']}
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/pulp3_migration.rake:31:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
Starting task.
2021-10-12 18:14:44 -0400: Content migration starting.
--------------------------------------------------------------------------------
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

  [content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-switchover"

Expected outcome:
The Pulp migration would succeed so we can upgrade to a newer versions of Foreman/Katello.

Foreman and Proxy versions:

[root@foreman foreman]# rpm -qa | egrep '(foreman|katello)'
foreman-dynflow-sidekiq-2.3.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
foreman.localdomain-tomcat-1.0-1.noarch
katello-repos-3.18.1-1.el7.noarch
foreman-installer-2.3.2-1.el7.noarch
foreman-selinux-2.3.2-1.el7.noarch
foreman-proxy-2.3.2-1.el7.noarch
foreman.localdomain-qpid-router-server-1.0-1.noarch
rubygem-foreman_maintain-0.7.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.3.0-1.el7.noarch
katello-common-3.18.1-1.el7.noarch
tfm-rubygem-foreman-tasks-3.0.3-1.fm2_3.el7.noarch
katello-3.18.1-1.el7.noarch
katello-service-3.13.4-1.el7.noarch
foreman-installer-katello-2.3.2-1.el7.noarch
tfm-rubygem-foreman_remote_execution-4.2.2-1.fm2_3.el7.noarch
katello-selinux-3.5.0-1.el7.noarch
foreman.localdomain-qpid-client-cert-1.0-1.noarch
foreman.localdomain-foreman-client-1.0-1.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
foreman.localdomain-foreman-proxy-client-1.0-1.noarch
tfm-rubygem-katello-3.18.1-1.el7.noarch
foreman-service-2.3.2-1.el7.noarch
foreman-cli-2.3.2-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
foreman-debug-2.3.2-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.24.2-1.el7.noarch
katello-server-ca-1.0-1.noarch
foreman.localdomain-apache-1.0-1.noarch
foreman.localdomain-foreman-proxy-1.0-1.noarch
foreman-release-scl-7-3.el7.noarch
foreman-postgresql-2.3.2-1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
katello-default-ca-1.0-1.noarch
tfm-rubygem-foreman_docker-5.0.0-3.fm2_1.el7.noarch
foreman.localdomain-qpid-router-client-1.0-1.noarch
katello-debug-3.18.1-1.el7.noarch
foreman-2.3.2-1.el7.noarch
foreman-vmware-2.3.2-1.el7.noarch
foreman.localdomain-qpid-broker-1.0-2.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
foreman.localdomain-puppet-client-1.0-1.noarch
foreman-release-2.3.2-1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch

Foreman and Proxy plugin versions:
See above (we only have the one Foreman server, but please let me know if anything isn’t in there and I will add it)

Distribution and version:

[root@foreman foreman]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)

Other relevant data:
I tried to run foreman-rake katello:delete_orphaned_content with no luck. I also tried the commands at Katello 3.17 to 3.18, cannot migrate to pulp3 - #35 by iballou also with no luck.

Please let me know what other information I can provide. Thank you in advance for any information you may be able to provide!

  • Chris M

Hey Chris,

Could you try upgrading to katello-3.18.3. The z stream contains a lot of migration related fixes and could solve the issue you’re seeing.

1 Like

Hi sajah,

Thank you for the suggestion. I ran into the qpid issue that was discussed in Issues with qpid-proton due to 0.35.0 release in EPEL but was able to get past it by adding the qpid-copr repository from https://github.com/theforeman/foreman-packaging/blob/rpm/3.0/packages/foreman/foreman-release/qpid-copr.repo

After a reboot I ran foreman-maintain content migration-stats and it failed with this output:

[root@foreman ~]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics:                       [FAIL]
Failed executing foreman-rake katello:pulp3_migration_stats, exit status 1:
 API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column katello_rpms.missing_from_migration does not exist
LINE 1: ... "katello_rpms"."migrated_pulp3_href" IS NULL AND "katello_r...
                                                             ^
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:656:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:489:in `select'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:70:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:77:in `select_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `block in exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:267:in `empty?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:279:in `any?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:56:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'

Caused by:
PG::UndefinedColumn: ERROR:  column katello_rpms.missing_from_migration does not exist
LINE 1: ... "katello_rpms"."migrated_pulp3_href" IS NULL AND "katello_r...
                                                             ^
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:656:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:489:in `select'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:70:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:77:in `select_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `block in exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:267:in `empty?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:279:in `any?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:56:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_stats
(See full trace by running task with --trace)
============Migration Summary================
Migrated/Total RPMs: 0/279295
Migrated/Total errata: 0/98460
Migrated/Total repositories: 0/300
Estimated migration time based on yum content: 4 hours, 25 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'

Note: ensure there is sufficient storage space for postgresql.
You will need additional space for your postgresql database.  The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
   will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
--------------------------------------------------------------------------------
Scenario [Retrieve Pulp 2 to Pulp 3 migration statistics] failed.

The following steps ended up in failing state:

  [content-migration-stats]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-migration-stats"

I’m guessing that is not terrible, as it also shows nothing having been migrated. Anyway, after that I ran into error ERF64-6496, and I found Foreman - ERF64-6496 [Foreman::MaintenanceException] after update

So I ran foreman-rake db:migrate:

[root@foreman ~]# foreman-rake db:migrate
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
== 20210201165835 AddMigrationMissingContent: migrating =======================
-- add_column("katello_rpms", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0134s
-- add_column("katello_rpms", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0006s
-- add_column("katello_module_streams", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0024s
-- add_column("katello_module_streams", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_errata", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0025s
-- add_column("katello_errata", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_package_groups", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0009s
-- add_column("katello_package_groups", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0021s
-- add_column("katello_yum_metadata_files", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0025s
-- add_column("katello_yum_metadata_files", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0010s
-- add_column("katello_srpms", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0025s
-- add_column("katello_srpms", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_files", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0021s
-- add_column("katello_files", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_docker_manifest_lists", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_docker_manifest_lists", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_docker_manifests", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0015s
-- add_column("katello_docker_manifests", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_docker_tags", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0010s
-- add_column("katello_docker_tags", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
-- add_column("katello_debs", :missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0032s
-- add_column("katello_debs", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
   -> 0.0004s
== 20210201165835 AddMigrationMissingContent: migrated (0.0412s) ==============

== 20210317170111 RemoveTheRemoteAddSetting: migrating ========================
== 20210317170111 RemoveTheRemoteAddSetting: migrated (0.0031s) ===============

== 20210420140050 AddPulp3HrefsToContentTypesDeb: migrating ===================
-- add_column("katello_debs", :migrated_pulp3_href, :string, {:default=>nil, :null=>true})
   -> 0.0006s
== 20210420140050 AddPulp3HrefsToContentTypesDeb: migrated (0.0007s) ==========

(I’m noticing the output of db:migrate appears to have created the column that the migration-stats command failed on)

That didn’t return an error but did prompt me to run foreman-rake apipie:cache which i did:

[root@foreman ~]# foreman-rake apipie:cache
2021-10-13 10:35:54 -0400 | Started
2021-10-13 10:35:57 -0400 | Documents loaded...
2021-10-13 10:35:57 -0400 | Processing docs for 
2021-10-13 10:36:10 -0400 | Processing docs for ko
2021-10-13 10:36:24 -0400 | Processing docs for en
2021-10-13 10:36:37 -0400 | Processing docs for nl_NL
2021-10-13 10:36:51 -0400 | Processing docs for fr
2021-10-13 10:37:06 -0400 | Processing docs for ja
2021-10-13 10:37:19 -0400 | Processing docs for gl
2021-10-13 10:37:33 -0400 | Processing docs for de
2021-10-13 10:37:47 -0400 | Processing docs for pt_BR
2021-10-13 10:38:01 -0400 | Processing docs for zh_TW
2021-10-13 10:38:15 -0400 | Processing docs for it
2021-10-13 10:38:29 -0400 | Processing docs for sv_SE
2021-10-13 10:38:43 -0400 | Processing docs for en_GB
2021-10-13 10:38:56 -0400 | Processing docs for pl
2021-10-13 10:39:10 -0400 | Processing docs for cs_CZ
2021-10-13 10:39:24 -0400 | Processing docs for zh_CN
2021-10-13 10:39:38 -0400 | Processing docs for ru
2021-10-13 10:39:53 -0400 | Processing docs for ca
2021-10-13 10:40:07 -0400 | Processing docs for es
2021-10-13 10:40:21 -0400 | Finished

which did not return an error either.

Then I ran foreman-rake db:seed which did not return an error.

[root@foreman ~]# foreman-rake db:seed
An admin user already exists, not seeding a new one.

For kicks I tried to get migration-stats again, and it did not fail this time!

[root@foreman ~]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: 
============Migration Summary================
Migrated/Total RPMs: 0/279279
Migrated/Total errata: 0/98460
Migrated/Total repositories: 0/300
Estimated migration time based on yum content: 4 hours, 25 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'

Note: ensure there is sufficient storage space for postgresql.
You will need additional space for your postgresql database.  The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
   will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
                                                                      [OK]
--------------------------------------------------------------------------------

At this point Foreman’s web interface was still unavailable, so I rebooted the VM (by the way, thank goodness for snapshots).

Now when I run foreman-maintain content prepare I’m getting this error:

[root@foreman ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Checking for valid Katello configuraton.
Starting task.
2021-10-13 10:49:39 -0400: Content migration starting. These steps may take a while to complete. Refer to `foreman-maintain content migration-stats` for an estimate.Migration failed, You will want to investigate: https://foreman.localdomain/foreman_tasks/tasks/11f7f26c-e406-48c9-a274-e4b0c21d4937
rake aborted!
ForemanTasks::TaskError: Task 11f7f26c-e406-48c9-a274-e4b0c21d4937: Katello::Errors::Pulp3Error: relation "pulp_2to3_migration_reposetup" does not exist
LINE 1: ...e", "pulp_2to3_migration_reposetup"."status" FROM "pulp_2to3...
                                                             ^
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration.rake:41:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

The 2 to 3 migration packages are installed (as far as I can tell):

[root@foreman ~]# rpm -qa | grep 2to3
tfm-rubygem-pulp_2to3_migration_client-0.11.0-1.el7.noarch
python3-pulp-2to3-migration-0.11.5-1.el7.noarch

Unfortunately this is where I’m stuck again, as I can’t find anything online about the pulp_2to3_migration_reposetup error above. It sounds like a database error, but re-running db:migrate and db:seed doesn’t fix it.

Looking at the logs it seems like there’s some more migrations that are needed. Does running foreman-installer help with those after the yum update?
https://theforeman.org/plugins/katello/3.18/upgrade/index.html