[Content-Migration] katello:correct_repositories and correct_puppet_environments fails blocking migration

Support
21

Justin,

I don’t have an /etc/foreman/katello.yaml. I only have an /etc/foreman/plugins/katello.yaml

[root@foreman plugins]# more katello.yaml
### File managed with puppet ###
## Module: puppet-katello

:katello:
  :rest_client_timeout: 3600

  :content_types:
    :file: true
    :yum: true
    :deb: false
    :puppet: true
    :docker: true
    :ostree: false

  :candlepin:
    :url: https://localhost:23443/candlepin
    :oauth_key: "katello"
    :oauth_secret: "redacted"
    :ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt

  :candlepin_events:
    :ssl_cert_file: /etc/pki/katello/certs/java-client.crt
    :ssl_key_file: /etc/pki/katello/private/java-client.key
    :ssl_ca_file: /etc/pki/katello/certs/katello-default-ca.crt


  :pulp:
    :url: https://foreman.solutionsathand.ca/pulp/api/v2/
    :ca_cert_file: /etc/pki/katello/certs/katello-server-ca.crt

  :use_pulp_2_for_content_type:
    :docker: false
    :file: false
    :yum: true
    :deb: false

  :katello_applicability: false

  # Internal configuration for communication from server to pulp crane service.
  :container_image_registry:
    :crane_url: https://foreman.solutionsathand.ca:5000
    :crane_ca_cert_file: /etc/pki/katello/certs/katello-server-ca.crt

Now, one thing we noticed earlier in the thread is that the upgrade routine is a bit fragile in that the deprecated types (for example puppet) had to be enabled in order to run. OSTree was disabled, so I may have to enable it. Will try that.

22

Enabled ostree and tried content prepare again, with the same issue.

[root@foreman plugins]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-04-15 20:29:01 -0400: Pre-migrating Pulp 2 ERRATUM content (general info) 2481/2884Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/afa08a76-9bee-4f5c-b062-35af2415a41b
rake aborted!
ForemanTasks::TaskError: Task afa08a76-9bee-4f5c-b062-35af2415a41b: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
23

Has there been any resolution on this? I am STILL having this issue.

Thanks

24

Still having this issue and still cannot migrate to pulp3. Any solution to this?

25

Did you try with latest foreman 2.3.x / katello 3.18.x packages ? aka katello 3.18.4 ?
I just did and migration works

1 Like
26

foreman-2.3.5-1.el7.noarch
katello-3.18.4-1.el7.noarch

[root@foreman] ~$ foreman-maintain content prepare

Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:

Rubocop not loaded.
Checking for valid Katello configuraton.
Starting task.
2021-08-03 14:04:02 -0400: Pre-migrating Pulp 2 package_group content (general info) 0/1471Migration failed, You will want to investigate: https://foreman.domain.com/foreman_tasks/tasks/2b880cba-097c-41b1-a291-beff30549f5f
rake aborted!
ForemanTasks::TaskError: Task 2b880cba-097c-41b1-a291-beff30549f5f: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
27

Hello, sorry for the dumb question, but after having upgraded packages, did you follow upgrade process, i.e

  • foreman rake upgrade checks are ok ?
  • invoke foreman-installer ?
    Are the repos synched successfully & all ?
28

Yes I have been through all of these steps for almost a year now and STILL have issues. I welcome your suggestions and yes, all of these checks pass. Thank you for trying though. :slight_smile:

29

I had some issues with migration that were resolved by upgrading to latest 3.18.x, running a reset on the migration and then rerunning it. Not the same error as i had, but might be worth a shot.
if i remember correctly the reset command is:
foreman-maintain content migration-reset

otherwise foreman-maintain content --help should show the command

Just my 2 cents, hope you get this fixed soon.

1 Like
30

Yeah thank you, I have tried that 100x too. Still doesn’t work in the end and I continue to get the same error. I am at the point of wanting to build a whole new Foreman server but I just don’t have the time for that right now. Thanks for your suggestion.

31

I’m still having the same basic issue, that I originally reported at the beginning of this thread.

[root@foreman iso]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Checking for valid Katello configuraton.
Starting task.
2021-08-10 17:24:55 -0400: Initial Migration steps complete.Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/2b04464e-3443-43c5-b237-df1f6070799a
rake aborted!
ForemanTasks::TaskError: Task 2b04464e-3443-43c5-b237-df1f6070799a: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Seems to be in the errata parts of the upgrade. I’ve been slowly deleting products and content to make it go away, but still not succeeding.

If there was a way of simply resetting the content without having to blow away the repo definitions, that would be acceptable. I’m rapidly approaching the point of simply deleting all products, content-views, etc. and trying to upgrade from there. It’s a good thing this is only home lab environment. I’m looking to build in parallel a new version at a work, and simply cutting over to that. The upgrade isn’t worth the headaches at this point in time.

1 Like
32

Did you try a migration reset and followed by a delete_orphaned_content before a new prepare?

33

I’ve tried it before, and just tried it again in exactly that order. No Joy. It’s definitely bailing on erratum.

[root@foreman iso]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Checking for valid Katello configuraton.
Starting task.
2021-08-11 22:28:15 -0400: Pre-migrating Pulp 2 erratum content (general info) 12004/12175Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/b9ea7165-e6c5-4608-b3d4-e684bfe97b2b
rake aborted!
ForemanTasks::TaskError: Task b9ea7165-e6c5-4608-b3d4-e684bfe97b2b: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

I watched it go, and noticed that the x/y counter was increasing for both, like it was finding new erratum to migrate. That wasn’t the same behaviour as the RPM content, for which the second number (out of total) didn’t change.

The actual task had this error message:

      {"message"=>"Migrating importers to Pulp 3",
       "code"=>"migrating.importers",
       "state"=>"failed",
       "total"=>13,
       "done"=>0}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/e34ee9ee-efc8-456f-aa12-f62a25b8dd5f/"],
    "reserved_resources_record"=>["pulp_2to3_migration"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/e34ee9ee-efc8-456f-aa12-f62a25b8dd5f/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>37, "failed"=>1}}
34

What exactly did you run? Did you wait for the delete orphans tasks to be finished? It is running in the background and can take while. Did migration stats show all zero after that?

35

Doesn’t the task view in foreman shows the pulp traceback?

You should be able to check for the exact error with curl on the foreman server:

# curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://foreman.solutionsathand.ca/pulp/api/v3/task-groups/e34ee9ee-efc8-456f-aa12-f62a25b8dd5f/' | python -m json.tool

This contains the href to the task in the tasks array, e.g.

...
    "tasks": [
        {
            "finished_at": "2021-04-27T20:02:42.454239Z",
            "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
            "pulp_created": "2021-04-27T17:19:50.099119Z",
            "pulp_href": "/pulp/api/v3/tasks/31fee624-f68b-4650-91fd-71fc7938266c/",
            "started_at": "2021-04-27T17:19:51.375034Z",
            "state": "failed",
            "worker": "/pulp/api/v3/workers/48993a1b-fbdc-4611-adfb-8bc53b7c8218/"
        }
...

Next adjust the curl URL above and use the pulp_href, e.g.

# curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://foreman.solutionsathand.ca/pulp/api/v3/tasks/31fee624-f68b-4650-91fd-71fc7938266c/' | python -m json.tool

There you should be an attribute name “error” which contains the description and traceback of the pulp task…

1 Like
36

Just to document the steps I’ve taken, and the output responses:

yum upgrade to current packages for Foreman 2.35 and Katello 3.18.4

foreman-installer to ensure the upgrade tasks are done.

foreman-rake katello:delete_orphaned_content

[root@foreman iso]# foreman-rake katello:pulp3_migration_stats
============Migration Summary================
Migrated/Total RPMs: 0/36233
Migrated/Total errata: 0/58303
Migrated/Total repositories: 0/93

Estimated migration time based on yum content: 0 hours, 24 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with ‘du -sh /var/lib/pulp/published/’

foreman-maintain content migration-reset to ensure everything is at a good starting point.

foreman-maintain content prepare

which fails and results in a stack trace.

[root@foreman iso]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Checking for valid Katello configuraton.
Starting task.
2021-08-17 15:09:33 -0400: Pre-migrating Pulp 2 erratum content (general info) 11004/11702Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/50f3359c-5698-440f-abc0-8e3a0a19c04d
rake aborted!
ForemanTasks::TaskError: Task 50f3359c-5698-440f-abc0-8e3a0a19c04d: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Looking in the web UI, the copy/paste is below for the errors from the affected task, Note the failure at the importer_migrator.migrate_to_pulp3(pulp2importer) line. Any advice on fixing this would be greatly appreciated! Another possible option would be to reset all content without blowing the definitions away, and allowing for some form of upgrade and switchover that would work directly with pulp3. Thanks in advance for any assistance you can provide!

Action:
Actions::Pulp3::ContentMigration
Input:
{"services_checked"=>["pulp3"],
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>3,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/11ebe41f-c381-4be5-b44b-c082507d9155/",
    "pulp_created"=>"2021-08-17T19:05:04.333+00:00",
    "state"=>"failed",
    "name"=>"pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "started_at"=>"2021-08-17T19:05:04.538+00:00",
    "finished_at"=>"2021-08-17T19:09:39.694+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
       "    rv = job.perform()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
       "    self._result = self._execute()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
       "    return self.func(*self.args, **self.kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 80, in migrate_from_pulp2\n" +
       "    migrate_importers(plan)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 141, in migrate_importers\n" +
       "    remote, created = importer_migrator.migrate_to_pulp3(pulp2importer)\n",
      "description"=>"'NoneType' object has no attribute 'migrate_to_pulp3'"},
    "worker"=>"/pulp/api/v3/workers/d3304a61-2ae7-4647-aa97-c28575b18eb0/",
    "child_tasks"=>[],
    "task_group"=>
     "/pulp/api/v3/task-groups/fca2ec9d-4b15-4d8d-9598-c7ef3529efef/",
    "progress_reports"=>
     [{"message"=>"Processing Pulp 2 repositories, importers, distributors",
       "code"=>"processing.repositories",
       "state"=>"completed",
       "total"=>93,
       "done"=>93},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>17,
       "done"=>17},
      {"message"=>"Pre-migrating Pulp 2 distribution content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>1,
       "done"=>1},
      {"message"=>"Pre-migrating Pulp 2 distribution content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>1,
       "done"=>1},
      {"message"=>"Pre-migrating Pulp 2 rpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>36233,
       "done"=>36233},
      {"message"=>"Pre-migrating Pulp 2 rpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>36233,
       "done"=>36233},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>12360,
       "done"=>12360},
      {"message"=>"Pre-migrating Pulp 2 erratum content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>12360,
       "done"=>12360},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>17,
       "done"=>17},
      {"message"=>"Pre-migrating Pulp 2 modulemd content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 modulemd content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>1,
       "done"=>1},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>1,
       "done"=>1},
      {"message"=>"Pre-migrating Pulp 2 package_group content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>256,
       "done"=>256},
      {"message"=>"Pre-migrating Pulp 2 package_group content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>256,
       "done"=>256},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>25,
       "done"=>25},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>25,
       "done"=>25},
      {"message"=>"Creating repositories in Pulp 3",
       "code"=>"creating.repositories",
       "state"=>"completed",
       "total"=>18,
       "done"=>18},
      {"message"=>"Migrating importers to Pulp 3",
       "code"=>"migrating.importers",
       "state"=>"failed",
       "total"=>13,
       "done"=>3}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/fca2ec9d-4b15-4d8d-9598-c7ef3529efef/"],
    "reserved_resources_record"=>["pulp_2to3_migration"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/fca2ec9d-4b15-4d8d-9598-c7ef3529efef/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>37, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Action:
Actions::Pulp3::ImportMigration
Input:
{"reimport_all"=>false,
 "dependency"=>Step(2).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>3,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{}
37

So, I’ve gotten tired of futzing with this, so I deleted all content views, products and repos. Deleted orphans, migration-reset, and then did migration stats (verifying 0/0), and then content prepare. It’s still failing with the importers.

[root@foreman iso]# foreman-rake katello:pulp3_migration_stats
============Migration Summary================
Migrated/Total RPMs: 0/0
Migrated/Total errata: 0/0
Migrated/Total repositories: 0/0

Estimated migration time based on yum content: fewer than 5 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
[root@foreman iso]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Checking for valid Katello configuraton.
Starting task.
2021-08-18 14:01:52 -0400: Content migration starting. Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/3d96172c-01a4-4238-9ee1-f6c75b0087c7
rake aborted!
ForemanTasks::TaskError: Task 3d96172c-01a4-4238-9ee1-f6c75b0087c7: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

The WebUI indicates the following error message:

Action:
Actions::Pulp3::ContentMigration
Input:
{"services_checked"=>["pulp3"],
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>3,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/4a19ef66-3770-4bf2-895e-a6fadd0a3119/",
    "pulp_created"=>"2021-08-18T18:01:52.844+00:00",
    "state"=>"failed",
    "name"=>"pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "started_at"=>"2021-08-18T18:01:53.018+00:00",
    "finished_at"=>"2021-08-18T18:01:53.858+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
       "    rv = job.perform()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
       "    self._result = self._execute()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
       "    return self.func(*self.args, **self.kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 80, in migrate_from_pulp2\n" +
       "    migrate_importers(plan)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 141, in migrate_importers\n" +
       "    remote, created = importer_migrator.migrate_to_pulp3(pulp2importer)\n",
      "description"=>"'NoneType' object has no attribute 'migrate_to_pulp3'"},
    "worker"=>"/pulp/api/v3/workers/d3304a61-2ae7-4647-aa97-c28575b18eb0/",
    "child_tasks"=>[],
    "task_group"=>
     "/pulp/api/v3/task-groups/31c2bf82-668b-4e7d-bdb6-7cb1ddafa01c/",
    "progress_reports"=>
     [{"message"=>"Processing Pulp 2 repositories, importers, distributors",
       "code"=>"processing.repositories",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 rpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 rpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 modulemd content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 modulemd content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Creating repositories in Pulp 3",
       "code"=>"creating.repositories",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Migrating importers to Pulp 3",
       "code"=>"migrating.importers",
       "state"=>"failed",
       "total"=>3,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 package_group content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 package_group content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/31c2bf82-668b-4e7d-bdb6-7cb1ddafa01c/"],
    "reserved_resources_record"=>["pulp_2to3_migration"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/31c2bf82-668b-4e7d-bdb6-7cb1ddafa01c/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>2, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Action:
Actions::Pulp3::ImportMigration
Input:
{"reimport_all"=>false,
 "dependency"=>Step(2).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>3,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{}

So I’m looking to just force the switch to pulp3 since I have no content remaining. Hoping that future versions will houseclean any incompatible configuration…

38

foreman-rake katello:pulp3_content_switchover

foreman-rake katello:pulp3_post_migration_check

both succeeded despite the previous importers failure. Rolling forward with Foreman & Katello upgrades before I recreate all my products and stuff…

39

Sadly I think I am going to need to do this too. I will probably get it done this weekend. Sigh…

Thanks for posting about this! Glad I am not the only one struggling with this.

40

@metalcated just be aware, the importers issue continued to be a thorn in my side even after moving forward. I went to Foreman 2.4 and Katello 4.0 latest point releases, and tried to migrate content or otherwise switch over to pulp3. No joy. I kept having errors, and tried a bunch of things. Decided to disable the built-in capsule content management, and was able to run foreman-installer without issue along with foreman-rake upgrade:run. I rolled forward again to Foreman 2.5.2 and Katello 4.1.2 successfully with the disabled internal capsule.

I tried re-enabling the capsule, more problems. There was just no way, even with all products and content deleted to get it working with the broken importers step, and there was no way to simply force a clean configuration for pulp 3. These issues had delayed me for some pipeline work I’m doing proof-of-concepts on, so I just decided to go nuclear, and ran foreman-installer --reset-data.

NOTE: That will drop all databases!!! Make sure you have copies of all your configuration; it wipes out everything, including external authentication setups, users, systems, reports; everything. Clean slate.

Now, that isn’t in itself a bad thing… It’s let me houseclean some cruft out, but now I am having to reconfigure everything. I do a lot of the config with bash scripts and hammer commands for now, so with some updates for the latest hammer configurations, I’m getting back to where I was pretty fast. I do plan to switch over to the foreman ansible modules in the future though…

So, all I can suggest is to make sure you backup everything you may want to reload, and make good notes on your current configuration. When I replace the older foreman system at work, I’m not going to bother with upgrading it; I’ll simply build a new one in parallel, bring it fully online and populated with content, and then use remote execution on the old one to repoint the client systems to the new one. It’ll be a lot more straight forward.

Hope that helps!

/Mike