Pulp3 Migration failed

Problem:
During migration to Pulp3:
foreman-installer --katello-enable-deb=false
chmod -R g+rwX /var/lib/pulp/content
find /var/lib/pulp/content -type d -perm -g-s -exec sudo chmod g+s {} ;
chgrp -R pulp /var/lib/pulp/content
foreman-maintain content migration-stats
foreman-maintain content prepare
foreman-maintain content switchover
Have an error:
error (RuntimeError): Cannot find repository type deb, is it enabled?

Expected outcome:
Successful migration
Foreman and Proxy versions:

Foreman and Proxy plugin versions:
Foreman release 2.3
Katello 3.18
Distribution and version:

Other relevant data:

Hi @el1psis,

The error is happening because the deb content type is disabled, but deb is set to use Pulp 2 instead of Pulp 3. Looks like it’s a bug.

To work around the issue, run the foreman-installer with --katello-use-pulp-2-for-deb false. Then, you should be able to run the switchover. Make sure you’re running this command only if you have no Pulp 2 deb content in your Katello environment. I’m assuming that is the case since you disabled the deb content type.

Hi @iballou,

Thank you for response.
I have preconfigured with --katello-use-pulp-2-for-deb=false.
But there is new issue:

Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
enabled
Starting task.
2021-02-24 14:26:42 +0600: Importing migrated content type rpm: 1440/1755Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_rpms record has migrated_pulp3_href NULL value
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.
The following steps ended up in failing state:
[content-switchover]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

@el1psis can you try running foreman-rake katello:delete_orphaned_content and then re-run content prepare? After that, you can run the following in the foreman console (foreman-rake console) to check if you still have NULL migrated_pulp3_hrefs:

::Katello::Rpm.where(migrated_pulp3_href: nil)

If you still see nil ones, trying reimport_all=true foreman-rake katello:pulp3_migration might help. That runs the Pulp 3 migration again but reimports everything.

1 Like

@iballou I have run

foreman-rake katello:delete_orphaned_content - successful, some output has been occured, like “Orphaned content dleltion started”

than:

foreman-maintain content prepare - successful

(foreman-rake console ) ::Katello::Rpm.where(migrated_pulp3_href: nil)


output:

Loading production environment (Rails 6.0.3.4)
irb(main):001:0> ::Katello::Rpm.where(migrated_pulp3_href: nil)
=> #<ActiveRecord::Relation [#<Katello::Rpm id: 83402, pulp_id: "34e70dcf-9455-404e-8b50-d6a972fa75bd", created_at: "2020-08-17 17:12:55", updated_at: "2020-08-17 17:12:55", name: "qpid-proton-cpp-docs", version: "0.31.0", release: "3.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-cpp-docs-0.31.0-3.el7.noarch.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "d203c6f5eff0bcbd47a0384b24343253c386abaeaa5109d656...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Documentation for the C++ development libraries fo...", nvra: "qpid-proton-cpp-docs-0.31.0-3.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 83405, pulp_id: "892c554e-2da4-49bc-9872-cc89b58120c1", created_at: "2020-08-17 17:13:01", updated_at: "2020-08-17 17:13:01", name: "rubygem-qpid_proton", version: "0.31.0", release: "3.el7", arch: "x86_64", epoch: "0", filename: "rubygem-qpid_proton-0.31.0-3.el7.x86_64.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "45365b13d7a2616e44880ff8a77f251a59253a41ace0f2b252...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Ruby language bindings for the Qpid Proton messagi...", nvra: "rubygem-qpid_proton-0.31.0-3.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 83407, pulp_id: "a1ffaf96-33d8-4bff-8121-162698fa4020", created_at: "2020-08-17 17:13:05", updated_at: "2020-08-17 17:13:05", name: "qpid-proton-c-docs", version: "0.31.0", release: "3.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-c-docs-0.31.0-3.el7.noarch.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "cc6983c17dd9728ab676f9d2affa46b13d02c0aea44ae0c88a...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Documentation for the C development libraries for ...", nvra: "qpid-proton-c-docs-0.31.0-3.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 83408, pulp_id: "c1334551-b576-494f-917a-a05c9d711fdb", created_at: "2020-08-17 17:13:07", updated_at: "2020-08-17 17:13:07", name: "python36-qpid-proton", version: "0.31.0", release: "3.el7", arch: "x86_64", epoch: "0", filename: "python36-qpid-proton-0.31.0-3.el7.x86_64.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "27ff6cf224ce3f7e0eaa7f02206af8a7f7e9f03a246ecfeb50...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Python language bindings for the Qpid Proton messa...", nvra: "python36-qpid-proton-0.31.0-3.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 86230, pulp_id: "111ce645-556b-43cc-9f16-7d5925b53172", created_at: "2020-10-23 03:28:10", updated_at: "2020-10-23 03:28:10", name: "qpid-proton-c-devel", version: "0.32.0", release: "2.el7", arch: "x86_64", epoch: "0", filename: "qpid-proton-c-devel-0.32.0-2.el7.x86_64.rpm", sourcerpm: "qpid-proton-0.32.0-2.el7.src.rpm", checksum: "7977ebdb913ff7faf74140a11b7d33d15963a6f278d41c3666...", version_sortable: "01-0.02-32.01-0", release_sortable: "01-2.$el.01-7", summary: "Development libraries for writing messaging apps w...", nvra: "qpid-proton-c-devel-0.32.0-2.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(32,)\"\",\"\"(0,)\"\"}\",\"{\"\"(2,)\"\",\"\"(0...">, #<Katello::Rpm id: 86239, pulp_id: "2e5dac78-3c40-4106-8209-2e8c93f7264d", created_at: "2020-10-23 03:28:13", updated_at: "2020-10-23 03:28:13", name: "yamllint", version: "1.25.0", release: "1.el7", arch: "noarch", epoch: "0", filename: "yamllint-1.25.0-1.el7.noarch.rpm", sourcerpm: "yamllint-1.25.0-1.el7.src.rpm", checksum: "143f5f4f36b6f79212e4c09e2ac67c0a122f1bfb6ceafa3975...", version_sortable: "01-1.02-25.01-0", release_sortable: "01-1.$el.01-7", summary: "A linter for YAML files", nvra: "yamllint-1.25.0-1.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(1,)\"\",\"\"(25,)\"\",\"\"(0,)\"\"}\",\"{\"\"(1,)\"\",\"\"(0...">, #<Katello::Rpm id: 86281, pulp_id: "ade9c25e-92b0-4e66-a4d0-1874eeb3cfd8", created_at: "2020-10-23 03:28:24", updated_at: "2020-10-23 03:28:24", name: "python36-bloom", version: "0.10.0", release: "1.el7", arch: "noarch", epoch: "0", filename: "python36-bloom-0.10.0-1.el7.noarch.rpm", sourcerpm: "python-bloom-0.10.0-1.el7.src.rpm", checksum: "167820140e2077d1b70bd55b7cafa859a7fcb288b196894ce1...", version_sortable: "01-0.02-10.01-0", release_sortable: "01-1.$el.01-7", summary: "HTML documentation for 'python-bloom'", nvra: "python36-bloom-0.10.0-1.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(10,)\"\",\"\"(0,)\"\"}\",\"{\"\"(1,)\"\",\"\"(0...">, #<Katello::Rpm id: 86297, pulp_id: "f0a734e3-d97c-4f74-9098-8667a351d8ef", created_at: "2020-10-23 03:28:29", updated_at: "2020-10-23 03:28:29", name: "qpid-proton-cpp-docs", version: "0.32.0", release: "2.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-cpp-docs-0.32.0-2.el7.noarch.rpm", sourcerpm: "qpid-proton-0.32.0-2.el7.src.rpm", checksum: "1de4f050ce259b1c40e859ad448eb5fde44f3a4e639a75d91c...", version_sortable: "01-0.02-32.01-0", release_sortable: "01-2.$el.01-7", summary: "Documentation for the C++ development libraries fo...", nvra: "qpid-proton-cpp-docs-0.32.0-2.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(32,)\"\",\"\"(0,)\"\"}\",\"{\"\"(2,)\"\",\"\"(0...">, #<Katello::Rpm id: 86296, pulp_id: "eff1e00d-9c53-45d3-a046-7fa82c93a886", created_at: "2020-10-23 03:28:29", updated_at: "2020-10-23 03:28:29", name: "qpid-proton-c-docs", version: "0.32.0", release: "2.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-c-docs-0.32.0-2.el7.noarch.rpm", sourcerpm: "qpid-proton-0.32.0-2.el7.src.rpm", checksum: "5af18312e83da9c2d0a861f9427a889e5147f7600950acd833...", version_sortable: "01-0.02-32.01-0", release_sortable: "01-2.$el.01-7", summary: "Documentation for the C development libraries for ...", nvra: "qpid-proton-c-docs-0.32.0-2.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(32,)\"\",\"\"(0,)\"\"}\",\"{\"\"(2,)\"\",\"\"(0...">, #<Katello::Rpm id: 87364, pulp_id: "b5a333ce-81f1-4ca9-af28-689e1f0ac497", created_at: "2020-11-12 17:11:26", updated_at: "2020-11-12 17:11:26", name: "batctl", version: "2020.4", release: "1.el7", arch: "x86_64", epoch: "0", filename: "batctl-2020.4-1.el7.x86_64.rpm", sourcerpm: "batctl-2020.4-1.el7.src.rpm", checksum: "500ed9487caa58a197673af882bdd96357e921b73db810f668...", version_sortable: "04-2020.01-4", release_sortable: "01-1.$el.01-7", summary: "B.A.T.M.A.N. advanced control and management tool", nvra: "batctl-2020.4-1.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(2020,)\"\",\"\"(4,)\"\"}\",\"{\"\"(1,)\"\",\"\"(0,el)\"\",...">, ...]>

As I understood this is not nil output.
After that I have rerun:

foreman-maintain content switchover

same output:

Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
enabled
Starting task.
2021-02-24 14:26:42 +0600: Importing migrated content type rpm: 1440/1755Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_rpms record has migrated_pulp3_href NULL value
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.
The following steps ended up in failing state:
[content-switchover]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”`

Hi @el1psis ,

The fact that your foreman-console command returned something means you do still have the nil migrated_pulp3_href. Can you try:

reimport_all=true foreman-rake katello:pulp3_migration
1 Like

Hi @iballou,

Command reimport_all=true foreman-rake katello:pulp3_migration was done successful:

Content Migration completed successfully

Is this successful migration to Pulp 3, or need to run foreman-maintain content switchover?
In web console is still Pulp2:
Pulp
Version 2.1.0
Pulp server version 2.21.5
Pulpcore
Version 2.1.0

at least has run foreman-rake katello:pulp3_content_switchover
output: ERROR: at least one katello_rpms record has migrated_pulp3_href NULL value

Its possible that some package was corrupted or missing on the filesystem. Katello 3.18.2 will feature a workflow designed to handle this situation and allow you to skip these during the switchover.

I’d hold off until katello 3.18.2, and then at that point:

  • re-run the ‘content prepare’ command
  • run:
    foreman-maintain content migration-stats
    

to analyze what is currently missing or corrupted. It will then give you steps to take to resolve it.