Hi sajah,
Thank you for the suggestion. I ran into the qpid issue that was discussed in Issues with qpid-proton due to 0.35.0 release in EPEL but was able to get past it by adding the qpid-copr repository from https://github.com/theforeman/foreman-packaging/blob/rpm/3.0/packages/foreman/foreman-release/qpid-copr.repo
After a reboot I ran foreman-maintain content migration-stats and it failed with this output:
[root@foreman ~]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: [FAIL]
Failed executing foreman-rake katello:pulp3_migration_stats, exit status 1:
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column katello_rpms.missing_from_migration does not exist
LINE 1: ... "katello_rpms"."migrated_pulp3_href" IS NULL AND "katello_r...
^
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:656:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:489:in `select'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:70:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:77:in `select_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `block in exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:267:in `empty?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:279:in `any?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:56:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Caused by:
PG::UndefinedColumn: ERROR: column katello_rpms.missing_from_migration does not exist
LINE 1: ... "katello_rpms"."migrated_pulp3_href" IS NULL AND "katello_r...
^
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:656:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:489:in `select'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:70:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:77:in `select_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `block in exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:317:in `exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:267:in `empty?'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:279:in `any?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:56:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration_stats.rake:55:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_stats
(See full trace by running task with --trace)
============Migration Summary================
Migrated/Total RPMs: 0/279295
Migrated/Total errata: 0/98460
Migrated/Total repositories: 0/300
Estimated migration time based on yum content: 4 hours, 25 minutes
Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
Note: ensure there is sufficient storage space for postgresql.
You will need additional space for your postgresql database. The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
--------------------------------------------------------------------------------
Scenario [Retrieve Pulp 2 to Pulp 3 migration statistics] failed.
The following steps ended up in failing state:
[content-migration-stats]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-migration-stats"
I’m guessing that is not terrible, as it also shows nothing having been migrated. Anyway, after that I ran into error ERF64-6496, and I found Foreman - ERF64-6496 [Foreman::MaintenanceException] after update
So I ran foreman-rake db:migrate:
[root@foreman ~]# foreman-rake db:migrate
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
== 20210201165835 AddMigrationMissingContent: migrating =======================
-- add_column("katello_rpms", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0134s
-- add_column("katello_rpms", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0006s
-- add_column("katello_module_streams", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0024s
-- add_column("katello_module_streams", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_errata", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0025s
-- add_column("katello_errata", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_package_groups", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0009s
-- add_column("katello_package_groups", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0021s
-- add_column("katello_yum_metadata_files", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0025s
-- add_column("katello_yum_metadata_files", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0010s
-- add_column("katello_srpms", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0025s
-- add_column("katello_srpms", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_files", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0021s
-- add_column("katello_files", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_docker_manifest_lists", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_docker_manifest_lists", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_docker_manifests", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0015s
-- add_column("katello_docker_manifests", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_docker_tags", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0010s
-- add_column("katello_docker_tags", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
-- add_column("katello_debs", :missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0032s
-- add_column("katello_debs", :ignore_missing_from_migration, :bool, {:default=>false, :null=>false})
-> 0.0004s
== 20210201165835 AddMigrationMissingContent: migrated (0.0412s) ==============
== 20210317170111 RemoveTheRemoteAddSetting: migrating ========================
== 20210317170111 RemoveTheRemoteAddSetting: migrated (0.0031s) ===============
== 20210420140050 AddPulp3HrefsToContentTypesDeb: migrating ===================
-- add_column("katello_debs", :migrated_pulp3_href, :string, {:default=>nil, :null=>true})
-> 0.0006s
== 20210420140050 AddPulp3HrefsToContentTypesDeb: migrated (0.0007s) ==========
(I’m noticing the output of db:migrate appears to have created the column that the migration-stats command failed on)
That didn’t return an error but did prompt me to run foreman-rake apipie:cache which i did:
[root@foreman ~]# foreman-rake apipie:cache
2021-10-13 10:35:54 -0400 | Started
2021-10-13 10:35:57 -0400 | Documents loaded...
2021-10-13 10:35:57 -0400 | Processing docs for
2021-10-13 10:36:10 -0400 | Processing docs for ko
2021-10-13 10:36:24 -0400 | Processing docs for en
2021-10-13 10:36:37 -0400 | Processing docs for nl_NL
2021-10-13 10:36:51 -0400 | Processing docs for fr
2021-10-13 10:37:06 -0400 | Processing docs for ja
2021-10-13 10:37:19 -0400 | Processing docs for gl
2021-10-13 10:37:33 -0400 | Processing docs for de
2021-10-13 10:37:47 -0400 | Processing docs for pt_BR
2021-10-13 10:38:01 -0400 | Processing docs for zh_TW
2021-10-13 10:38:15 -0400 | Processing docs for it
2021-10-13 10:38:29 -0400 | Processing docs for sv_SE
2021-10-13 10:38:43 -0400 | Processing docs for en_GB
2021-10-13 10:38:56 -0400 | Processing docs for pl
2021-10-13 10:39:10 -0400 | Processing docs for cs_CZ
2021-10-13 10:39:24 -0400 | Processing docs for zh_CN
2021-10-13 10:39:38 -0400 | Processing docs for ru
2021-10-13 10:39:53 -0400 | Processing docs for ca
2021-10-13 10:40:07 -0400 | Processing docs for es
2021-10-13 10:40:21 -0400 | Finished
which did not return an error either.
Then I ran foreman-rake db:seed which did not return an error.
[root@foreman ~]# foreman-rake db:seed
An admin user already exists, not seeding a new one.
For kicks I tried to get migration-stats again, and it did not fail this time!
[root@foreman ~]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics:
============Migration Summary================
Migrated/Total RPMs: 0/279279
Migrated/Total errata: 0/98460
Migrated/Total repositories: 0/300
Estimated migration time based on yum content: 4 hours, 25 minutes
Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
Note: ensure there is sufficient storage space for postgresql.
You will need additional space for your postgresql database. The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
[OK]
--------------------------------------------------------------------------------
At this point Foreman’s web interface was still unavailable, so I rebooted the VM (by the way, thank goodness for snapshots).
Now when I run foreman-maintain content prepare I’m getting this error:
[root@foreman ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Checking for valid Katello configuraton.
Starting task.
2021-10-13 10:49:39 -0400: Content migration starting. These steps may take a while to complete. Refer to `foreman-maintain content migration-stats` for an estimate.Migration failed, You will want to investigate: https://foreman.localdomain/foreman_tasks/tasks/11f7f26c-e406-48c9-a274-e4b0c21d4937
rake aborted!
ForemanTasks::TaskError: Task 11f7f26c-e406-48c9-a274-e4b0c21d4937: Katello::Errors::Pulp3Error: relation "pulp_2to3_migration_reposetup" does not exist
LINE 1: ...e", "pulp_2to3_migration_reposetup"."status" FROM "pulp_2to3...
^
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration.rake:41:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
The 2 to 3 migration packages are installed (as far as I can tell):
[root@foreman ~]# rpm -qa | grep 2to3
tfm-rubygem-pulp_2to3_migration_client-0.11.0-1.el7.noarch
python3-pulp-2to3-migration-0.11.5-1.el7.noarch
Unfortunately this is where I’m stuck again, as I can’t find anything online about the pulp_2to3_migration_reposetup error above. It sounds like a database error, but re-running db:migrate and db:seed doesn’t fix it.
