Pulp2 to Pulp3 migration fails

**Problem: Pulp2 to Pulp3 migration fails at switchover

**Expected outcome:Pulp2 to Pulp3 migration works

**Foreman and Proxy versions:2.3.5 / 2.3.5

Foreman and Proxy plugin versions:

**Distribution and version: CentOS 7

Other relevant data:

Hi,

I’m still stuck on Katello 3.18 and trying to upgrade to 4.0. In advance I’m trying to migrate from pulp2 to pulp3. I managed to get pass the

foreman-maintain content prepare

by accepting 3000+ corrupted or missing rpm packages. I have given up on fixing them.

Now I ran the switchover

foreman-maintain content switchover

and it fails with an error of:

[root@katello ~]# foreman-rake katello:pulp3_migration --trace
Rubocop not loaded.
** Invoke katello:pulp3_migration (first_time)
** Invoke dynflow:client (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute dynflow:client
** Invoke katello:check_config (first_time)
** Invoke environment
** Execute katello:check_config
Checking for valid Katello configuraton.
** Execute katello:pulp3_migration
Starting task.
2022-06-29 13:27:45 +0000: Starting katello import phase.Migration failed, You will want to investigate: https://katello.a.space.corp/foreman_tasks/tasks/3733fdb3-1ca3-4401-9a0a-65e6e9305040
rake aborted!
ForemanTasks::TaskError: Task 3733fdb3-1ca3-4401-9a0a-65e6e9305040: Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 502
Response headers: {"date"=>"Wed, 29 Jun 2022 13:27:17 GMT", "server"=>"Apache", "content-length"=>"439", "connection"=>"close", "content-type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/pulp/api/v3/pulp2repositories/">GET&nbsp;/pulp/api/v3/pulp2repositories/</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration.rake:41:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:235:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:pulp3_migration

Google didn’t come up with any similar errors so I thought I answer this question here. Any help would be highly appreciated.

Best Regards,
Oliver

It sounds like perhaps Pulp 3 is offline when it is expected to be online?

Hi,

I’m currently deleting all the content that is listed as broken. Once that is done I will try to reset the migration and start from scratch. I noticed in the logs that mongodb hat an out of memory error and so I increased the RAM of the VM running katello to 32GB. At least that error did not come up again.

Hi again,

I think I’m getting closer. I have now removed all the broken content but the prepare step still fails:

> [root@katello ~]# foreman-maintain health check
> Running ForemanMaintain::Scenario::FilteredScenario
> ================================================================================
> Check number of fact names in database:                               [OK]
> --------------------------------------------------------------------------------
> Check whether all services are running:                               [OK]
> --------------------------------------------------------------------------------
> Check whether all services are running using the ping call:           [OK]
> --------------------------------------------------------------------------------
> Check for paused tasks:                                               [OK]
> --------------------------------------------------------------------------------
> Check to verify no empty CA cert requests exist:                      [OK]
> --------------------------------------------------------------------------------
> [root@katello ~]# foreman-maintain content migration-stats
> Running Retrieve Pulp 2 to Pulp 3 migration statistics
> ================================================================================
> Retrieve Pulp 2 to Pulp 3 migration statistics:
> Rubocop not loaded.
> ============Migration Summary================
> Migrated/Total RPMs: 105642/105655
> Migrated/Total errata: 4767/4771
> Migrated/Total repositories: 63/63
> Estimated migration time based on yum content: 0 hours, 6 minutes
> 
> Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
> Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
> 
> Note: ensure there is sufficient storage space for postgresql.
> You will need additional space for your postgresql database.  The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
>    will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
>                                                                       [OK]
> --------------------------------------------------------------------------------
[root@katello ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Rubocop not loaded.
Checking for valid Katello configuraton.
Starting task.
2022-07-06 15:18:20 +0000: Pre-migrating Pulp 2 erratum content (general info) 105182/105275Migration failed, You will want to investigate: https://katello.a.space.corp/foreman_tasks/tasks/63819259-285f-43e9-9895-7c5cdb3f66c2
rake aborted!
ForemanTasks::TaskError: Task 63819259-285f-43e9-9895-7c5cdb3f66c2: Katello::Errors::Pulp3Error: '705bbd16-6f68-4279-a51b-8bae4aa5a660'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration.rake:41:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

I see this error in /var/log/messages:

Jul  6 15:18:18 katello pulpcore-worker-6: /usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1427: RuntimeWarning: DateTimeField Pulp2RepoContent.pulp2_created received a naive datetime (2022-07-05 12:47:54) while time zone support is active.
Jul  6 15:18:18 katello pulpcore-worker-6: RuntimeWarning)
Jul  6 15:18:18 katello pulpcore-worker-6: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jul  6 15:18:18 katello pulpcore-worker-6: rv = job.perform()
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jul  6 15:18:18 katello pulpcore-worker-6: self._result = self._execute()
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jul  6 15:18:18 katello pulpcore-worker-6: return self.func(*self.args, **self.kwargs)
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 77, in migrate_from_pulp2
Jul  6 15:18:18 katello pulpcore-worker-6: pre_migrate_all_content(plan)
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 70, in pre_migrate_all_content
Jul  6 15:18:18 katello pulpcore-worker-6: pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 305, in pre_migrate_content_type
Jul  6 15:18:18 katello pulpcore-worker-6: record = pulp2_content_by_id[relation.pulp2_unit_id]
Jul  6 15:18:18 katello pulpcore-worker-6: KeyError: '705bbd16-6f68-4279-a51b-8bae4aa5a660'
Jul  6 15:18:18 katello pulpcore-worker-6: Traceback (most recent call last):
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jul  6 15:18:18 katello pulpcore-worker-6: rv = job.perform()
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jul  6 15:18:18 katello pulpcore-worker-6: self._result = self._execute()
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jul  6 15:18:18 katello pulpcore-worker-6: return self.func(*self.args, **self.kwargs)
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 77, in migrate_from_pulp2
Jul  6 15:18:18 katello pulpcore-worker-6: pre_migrate_all_content(plan)
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 70, in pre_migrate_all_content
Jul  6 15:18:18 katello pulpcore-worker-6: pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
Jul  6 15:18:18 katello pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 305, in pre_migrate_content_type
Jul  6 15:18:18 katello pulpcore-worker-6: record = pulp2_content_by_id[relation.pulp2_unit_id]
Jul  6 15:18:18 katello pulpcore-worker-6: KeyError: '705bbd16-6f68-4279-a51b-8bae4aa5a660'
Jul  6 15:18:18 katello pulpcore-worker-6: pulp: rq.worker:INFO: 1124@katello.a.space.corp: abdffe57-46e7-49de-8dcb-2b8d85634b75
Jul  6 15:18:18 katello pulpcore-worker-6: pulp: rq.worker:INFO: 1124@katello.a.space.corp: Job OK (abdffe57-46e7-49de-8dcb-2b8d85634b75)
Jul  6 15:18:20 katello pulpcore-api: - - [06/Jul/2022:15:18:20 +0000] "GET /pulp/api/v3/tasks/c6f48248-c4c1-4509-afb1-5137b8ba40ab/ HTTP/1.1" 200 3969 "-" "OpenAPI-Generator/3.7.1/ruby"
Jul  6 15:18:20 katello pulpcore-api: - - [06/Jul/2022:15:18:20 +0000] "GET /pulp/api/v3/task-groups/3b87b700-bb21-4f68-8a23-cbccb817c07c/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"

The failed tasks in Katello WebUI only shows this:

‘705bbd16-6f68-4279-a51b-8bae4aa5a660’

I assume its a pulp repo???

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/c6f48248-c4c1-4509-afb1-5137b8ba40ab/",
    "pulp_created"=>"2022-07-06T15:09:11.381+00:00",
    "state"=>"failed",
    "name"=>"pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "started_at"=>"2022-07-06T15:09:11.601+00:00",
    "finished_at"=>"2022-07-06T15:18:18.269+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
       "    rv = job.perform()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
       "    self._result = self._execute()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
       "    return self.func(*self.args, **self.kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 77, in migrate_from_pulp2\n" +
       "    pre_migrate_all_content(plan)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 70, in pre_migrate_all_content\n" +
       "    pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 305, in pre_migrate_content_type\n" +
       "    record = pulp2_content_by_id[relation.pulp2_unit_id]\n",
      "description"=>"'705bbd16-6f68-4279-a51b-8bae4aa5a660'"},
    "worker"=>"/pulp/api/v3/workers/28146364-9e10-4171-9e77-091d88c977e4/",
    "child_tasks"=>[],
    "task_group"=>
     "/pulp/api/v3/task-groups/3b87b700-bb21-4f68-8a23-cbccb817c07c/",
    "progress_reports"=>
     [{"message"=>"Processing Pulp 2 repositories, importers, distributors",
       "code"=>"processing.repositories",
       "state"=>"completed",
       "total"=>5,
       "done"=>5},
      {"message"=>"Pre-migrating Pulp 2 deb content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 deb content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 deb_release content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 deb_release content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 deb_component content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 deb_component content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 rpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 rpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"running",
       "total"=>108460,
       "done"=>108419},
      {"message"=>"Pre-migrating Pulp 2 erratum content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"running",
       "total"=>108460,
       "done"=>108419}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/3b87b700-bb21-4f68-8a23-cbccb817c07c/"],
    "reserved_resources_record"=>["pulp_2to3_migration"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/3b87b700-bb21-4f68-8a23-cbccb817c07c/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>54, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: '705bbd16-6f68-4279-a51b-8bae4aa5a660'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

I will now do a reset and retry the prepare.

Running the reset and two more reboots i finally finished the prepare phase without any more errors:

[root@katello ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3

Prepare content for Pulp 3:
Rubocop not loaded.
Checking for valid Katello configuraton.
Starting task.
2022-07-06 22:49:53 +0000: Importing migrated content type package_group: 360/851
Content Migration completed successfully

Script started on Wed 06 Jul 2022 07:59:17 PM UTC
Rubocop not loaded.
Checking for valid Katello configuraton.
Starting task.
2022-07-06 22:49:53 +0000: Importing migrated content type package_group: 360/851
Content Migration completed successfully
[OK]

and I was able to successfully run the switchover:

[root@katello ~]# foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3

Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
Rubocop not loaded.
Checking for valid Katello configuraton.
Starting task.
2022-07-07 06:09:40 +0000: Initial Migration steps complete.
Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
Rubocop not loaded.
Switching specified content over to pulp 3
Rubocop not loaded.
Checking for valid Katello configuraton.
Content Switchover completed successfully
Re-running the installer to switch specified content over to pulp3 [OK]

Regenerate Apipie cache: [OK]

Great stuff :slight_smile:

Glad it finally worked out! Sorry for the trouble : /