Pulp3 Migration failed

el1psis · February 19, 2021, 4:37am

Problem:
During migration to Pulp3:
foreman-installer --katello-enable-deb=false
chmod -R g+rwX /var/lib/pulp/content
find /var/lib/pulp/content -type d -perm -g-s -exec sudo chmod g+s {} ;
chgrp -R pulp /var/lib/pulp/content
foreman-maintain content migration-stats
foreman-maintain content prepare
foreman-maintain content switchover
Have an error:
error (RuntimeError): Cannot find repository type deb, is it enabled?

Expected outcome:
Successful migration
Foreman and Proxy versions:

Foreman and Proxy plugin versions:
Foreman release 2.3
Katello 3.18
Distribution and version:

Other relevant data:

iballou · February 22, 2021, 10:03pm

Hi @el1psis,

The error is happening because the deb content type is disabled, but deb is set to use Pulp 2 instead of Pulp 3. Looks like it’s a bug.

To work around the issue, run the foreman-installer with --katello-use-pulp-2-for-deb false. Then, you should be able to run the switchover. Make sure you’re running this command only if you have no Pulp 2 deb content in your Katello environment. I’m assuming that is the case since you disabled the deb content type.

el1psis · February 24, 2021, 8:35am

Hi @iballou,

Thank you for response.
I have preconfigured with --katello-use-pulp-2-for-deb=false.
But there is new issue:

Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
enabled
Starting task.
2021-02-24 14:26:42 +0600: Importing migrated content type rpm: 1440/1755Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_rpms record has migrated_pulp3_href NULL value
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.
The following steps ended up in failing state:
[content-switchover]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

iballou · February 24, 2021, 10:16pm

@el1psis can you try running foreman-rake katello:delete_orphaned_content and then re-run content prepare? After that, you can run the following in the foreman console (foreman-rake console) to check if you still have NULL migrated_pulp3_hrefs:

::Katello::Rpm.where(migrated_pulp3_href: nil)

If you still see nil ones, trying reimport_all=true foreman-rake katello:pulp3_migration might help. That runs the Pulp 3 migration again but reimports everything.

el1psis · February 25, 2021, 3:36am

@iballou I have run

foreman-rake katello:delete_orphaned_content - successful, some output has been occured, like “Orphaned content dleltion started”

than:

foreman-maintain content prepare - successful

(foreman-rake console ) ::Katello::Rpm.where(migrated_pulp3_href: nil)

output:

Loading production environment (Rails 6.0.3.4)
irb(main):001:0> ::Katello::Rpm.where(migrated_pulp3_href: nil)
=> #<ActiveRecord::Relation [#<Katello::Rpm id: 83402, pulp_id: "34e70dcf-9455-404e-8b50-d6a972fa75bd", created_at: "2020-08-17 17:12:55", updated_at: "2020-08-17 17:12:55", name: "qpid-proton-cpp-docs", version: "0.31.0", release: "3.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-cpp-docs-0.31.0-3.el7.noarch.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "d203c6f5eff0bcbd47a0384b24343253c386abaeaa5109d656...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Documentation for the C++ development libraries fo...", nvra: "qpid-proton-cpp-docs-0.31.0-3.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 83405, pulp_id: "892c554e-2da4-49bc-9872-cc89b58120c1", created_at: "2020-08-17 17:13:01", updated_at: "2020-08-17 17:13:01", name: "rubygem-qpid_proton", version: "0.31.0", release: "3.el7", arch: "x86_64", epoch: "0", filename: "rubygem-qpid_proton-0.31.0-3.el7.x86_64.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "45365b13d7a2616e44880ff8a77f251a59253a41ace0f2b252...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Ruby language bindings for the Qpid Proton messagi...", nvra: "rubygem-qpid_proton-0.31.0-3.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 83407, pulp_id: "a1ffaf96-33d8-4bff-8121-162698fa4020", created_at: "2020-08-17 17:13:05", updated_at: "2020-08-17 17:13:05", name: "qpid-proton-c-docs", version: "0.31.0", release: "3.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-c-docs-0.31.0-3.el7.noarch.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "cc6983c17dd9728ab676f9d2affa46b13d02c0aea44ae0c88a...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Documentation for the C development libraries for ...", nvra: "qpid-proton-c-docs-0.31.0-3.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 83408, pulp_id: "c1334551-b576-494f-917a-a05c9d711fdb", created_at: "2020-08-17 17:13:07", updated_at: "2020-08-17 17:13:07", name: "python36-qpid-proton", version: "0.31.0", release: "3.el7", arch: "x86_64", epoch: "0", filename: "python36-qpid-proton-0.31.0-3.el7.x86_64.rpm", sourcerpm: "qpid-proton-0.31.0-3.el7.src.rpm", checksum: "27ff6cf224ce3f7e0eaa7f02206af8a7f7e9f03a246ecfeb50...", version_sortable: "01-0.02-31.01-0", release_sortable: "01-3.$el.01-7", summary: "Python language bindings for the Qpid Proton messa...", nvra: "python36-qpid-proton-0.31.0-3.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(31,)\"\",\"\"(0,)\"\"}\",\"{\"\"(3,)\"\",\"\"(0...">, #<Katello::Rpm id: 86230, pulp_id: "111ce645-556b-43cc-9f16-7d5925b53172", created_at: "2020-10-23 03:28:10", updated_at: "2020-10-23 03:28:10", name: "qpid-proton-c-devel", version: "0.32.0", release: "2.el7", arch: "x86_64", epoch: "0", filename: "qpid-proton-c-devel-0.32.0-2.el7.x86_64.rpm", sourcerpm: "qpid-proton-0.32.0-2.el7.src.rpm", checksum: "7977ebdb913ff7faf74140a11b7d33d15963a6f278d41c3666...", version_sortable: "01-0.02-32.01-0", release_sortable: "01-2.$el.01-7", summary: "Development libraries for writing messaging apps w...", nvra: "qpid-proton-c-devel-0.32.0-2.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(32,)\"\",\"\"(0,)\"\"}\",\"{\"\"(2,)\"\",\"\"(0...">, #<Katello::Rpm id: 86239, pulp_id: "2e5dac78-3c40-4106-8209-2e8c93f7264d", created_at: "2020-10-23 03:28:13", updated_at: "2020-10-23 03:28:13", name: "yamllint", version: "1.25.0", release: "1.el7", arch: "noarch", epoch: "0", filename: "yamllint-1.25.0-1.el7.noarch.rpm", sourcerpm: "yamllint-1.25.0-1.el7.src.rpm", checksum: "143f5f4f36b6f79212e4c09e2ac67c0a122f1bfb6ceafa3975...", version_sortable: "01-1.02-25.01-0", release_sortable: "01-1.$el.01-7", summary: "A linter for YAML files", nvra: "yamllint-1.25.0-1.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(1,)\"\",\"\"(25,)\"\",\"\"(0,)\"\"}\",\"{\"\"(1,)\"\",\"\"(0...">, #<Katello::Rpm id: 86281, pulp_id: "ade9c25e-92b0-4e66-a4d0-1874eeb3cfd8", created_at: "2020-10-23 03:28:24", updated_at: "2020-10-23 03:28:24", name: "python36-bloom", version: "0.10.0", release: "1.el7", arch: "noarch", epoch: "0", filename: "python36-bloom-0.10.0-1.el7.noarch.rpm", sourcerpm: "python-bloom-0.10.0-1.el7.src.rpm", checksum: "167820140e2077d1b70bd55b7cafa859a7fcb288b196894ce1...", version_sortable: "01-0.02-10.01-0", release_sortable: "01-1.$el.01-7", summary: "HTML documentation for 'python-bloom'", nvra: "python36-bloom-0.10.0-1.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(10,)\"\",\"\"(0,)\"\"}\",\"{\"\"(1,)\"\",\"\"(0...">, #<Katello::Rpm id: 86297, pulp_id: "f0a734e3-d97c-4f74-9098-8667a351d8ef", created_at: "2020-10-23 03:28:29", updated_at: "2020-10-23 03:28:29", name: "qpid-proton-cpp-docs", version: "0.32.0", release: "2.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-cpp-docs-0.32.0-2.el7.noarch.rpm", sourcerpm: "qpid-proton-0.32.0-2.el7.src.rpm", checksum: "1de4f050ce259b1c40e859ad448eb5fde44f3a4e639a75d91c...", version_sortable: "01-0.02-32.01-0", release_sortable: "01-2.$el.01-7", summary: "Documentation for the C++ development libraries fo...", nvra: "qpid-proton-cpp-docs-0.32.0-2.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(32,)\"\",\"\"(0,)\"\"}\",\"{\"\"(2,)\"\",\"\"(0...">, #<Katello::Rpm id: 86296, pulp_id: "eff1e00d-9c53-45d3-a046-7fa82c93a886", created_at: "2020-10-23 03:28:29", updated_at: "2020-10-23 03:28:29", name: "qpid-proton-c-docs", version: "0.32.0", release: "2.el7", arch: "noarch", epoch: "0", filename: "qpid-proton-c-docs-0.32.0-2.el7.noarch.rpm", sourcerpm: "qpid-proton-0.32.0-2.el7.src.rpm", checksum: "5af18312e83da9c2d0a861f9427a889e5147f7600950acd833...", version_sortable: "01-0.02-32.01-0", release_sortable: "01-2.$el.01-7", summary: "Documentation for the C development libraries for ...", nvra: "qpid-proton-c-docs-0.32.0-2.el7.noarch", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(0,)\"\",\"\"(32,)\"\",\"\"(0,)\"\"}\",\"{\"\"(2,)\"\",\"\"(0...">, #<Katello::Rpm id: 87364, pulp_id: "b5a333ce-81f1-4ca9-af28-689e1f0ac497", created_at: "2020-11-12 17:11:26", updated_at: "2020-11-12 17:11:26", name: "batctl", version: "2020.4", release: "1.el7", arch: "x86_64", epoch: "0", filename: "batctl-2020.4-1.el7.x86_64.rpm", sourcerpm: "batctl-2020.4-1.el7.src.rpm", checksum: "500ed9487caa58a197673af882bdd96357e921b73db810f668...", version_sortable: "04-2020.01-4", release_sortable: "01-1.$el.01-7", summary: "B.A.T.M.A.N. advanced control and management tool", nvra: "batctl-2020.4-1.el7.x86_64", modular: false, migrated_pulp3_href: nil, evr: "(0,\"{\"\"(2020,)\"\",\"\"(4,)\"\"}\",\"{\"\"(1,)\"\",\"\"(0,el)\"\",...">, ...]>

As I understood this is not nil output.
After that I have rerun:

foreman-maintain content switchover

same output:

Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
enabled
Starting task.
2021-02-24 14:26:42 +0600: Importing migrated content type rpm: 1440/1755Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_rpms record has migrated_pulp3_href NULL value
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.
The following steps ended up in failing state:
[content-switchover]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”`

iballou · February 25, 2021, 6:25pm

Hi @el1psis ,

The fact that your foreman-console command returned something means you do still have the nil migrated_pulp3_href. Can you try:

reimport_all=true foreman-rake katello:pulp3_migration

el1psis · February 26, 2021, 7:31am

Hi @iballou,

Command reimport_all=true foreman-rake katello:pulp3_migration was done successful:

Content Migration completed successfully

Is this successful migration to Pulp 3, or need to run foreman-maintain content switchover?
In web console is still Pulp2:
Pulp
Version 2.1.0
Pulp server version 2.21.5
Pulpcore
Version 2.1.0

el1psis · March 1, 2021, 6:24am

at least has run foreman-rake katello:pulp3_content_switchover
output: ERROR: at least one katello_rpms record has migrated_pulp3_href NULL value

Justin_Sherrill · March 1, 2021, 3:45pm

Its possible that some package was corrupted or missing on the filesystem. Katello 3.18.2 will feature a workflow designed to handle this situation and allow you to skip these during the switchover.

I’d hold off until katello 3.18.2, and then at that point:

re-run the ‘content prepare’ command

run:

foreman-maintain content migration-stats

to analyze what is currently missing or corrupted. It will then give you steps to take to resolve it.

el1psis · March 27, 2021, 4:24am

Hi @Justin_Sherrill,
Thank you for help.
I hold off until katello 3.18.2 has been released.
First I have rolled back to snapshot where i didnt run any migrations.
Than I try to do this:
foreman-installer --katello-enable-deb=false --katello-use-pulp-2-for-deb=false
chmod -R g+rwX /var/lib/pulp/content
find /var/lib/pulp/content -type d -perm -g-s -exec sudo chmod g+s {} ;
chgrp -R pulp /var/lib/pulp/content
foreman-maintain content migration-stats
foreman-maintain content prepare
output:

Prepare content for Pulp 3:
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Starting task.
2021-03-25 09:53:52 +0600: Initial Migration steps complete.
2021-03-25 10:00:53 +0600: Pre-migrating Pulp 2 RPM content (general info) 82300/106977
2021-03-25 10:08:24 +0600: Migrating rpm content to Pulp 3 rpm 2001/106977
2021-03-25 10:16:04 +0600: Migrating rpm content to Pulp 3 rpm 44860/106977
2021-03-25 10:27:25 +0600: Migrating rpm content to Pulp 3 rpm 79999/106977
2021-03-25 10:34:26 +0600: Migrating rpm content to Pulp 3 rpm 99886/106977
2021-03-25 10:41:57 +0600: Migrating rpm content to Pulp 3 erratum 39719/46297
2021-03-25 10:52:07 +0600: Repo version creation 25/35
2021-03-25 11:04:41 +0600: Distribution creation 55/57
2021-03-25 11:10:32 +0600: Importing migrated content type rpm: 9990/106977
2021-03-25 11:21:43 +0600: Importing migrated content type rpm: 51300/106977
2021-03-25 11:25:54 +0600: Importing migrated content type rpm: 66870/106977
2021-03-25 11:34:35 +0600: Importing migrated content type rpm: 96660/106977
2021-03-25 11:43:16 +0600: Importing migrated content type erratum: 38000/46297
2021-03-25 11:44:26 +0600: Importing migrated content type erratum: 48000/46297
Some corrupted or missing content found, run 'foreman-maintain content migration-stats' for more information.
                                                                  [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 255
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

foreman-maintain content migration-stats:

Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics:
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
============Migration Summary================
Migrated/Total RPMs: 104965/106977
Migrated/Total errata: 75794/75794
Migrated/Total repositories: 57/57

Estimated migration time based on yum content: fewer than 5 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'

============Missing/Corrupted Content Summary================
WARNING: MISSING OR CORRUPTED CONTENT DETECTED
Corrupted or Missing Rpm: 2012/106977
Corrupted or missing content has been detected, you can examine the list of content in /tmp/unmigratable_content-20210325-28817-9cyo58 and take action by either:
1. Performing a 'Verify Checksum' sync under Advanced Sync Options, let it complete, and re-running the migration
2. Deleting/disabling the affected repositories and running orphan cleanup (foreman-rake katello:delete_orphaned_content) and re-running the migration
3. Manually correcting files on the filesystem in /var/lib/pulp/content/ and re-running the migration
4. Mark currently corrupted or missing content as skipped (foreman-rake katello:approve_corrupted_migration_content).  This will skip migration of missing or corrupted content.
                                                                  [OK]
--------------------------------------------------------------------------------

Performing Verify Checksum didnt help, same output as there are corrupted or missing content has been detected
So at last I have run “foreman-rake katello:approve_corrupted_migration_content”

foreman-maintain content migration-stats
foreman-maintain content prepare
foreman-maintain content switchover – output:

Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Starting task.
021-03-27 10:05:18 +0600: Importing migrated content type rpm: 29340/31754
Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Switching specified content over to pulp 3                            [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
 API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
ERROR: at least one Erratum record has migrated_pulp3_href NULL value
--------------------------------------------------------------------------------
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

  [content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-switchover"

Please help, what I`m doing wrong?

Justin_Sherrill · April 14, 2021, 2:24pm

Apologies for the delay. I don’t know that you’ve done anything wrong, at least not recently

The migration identified 2012 rpms that either are a) missing from the filesystem b) corrupted on the filesystem or c) not in pulp

I would take a look at /tmp/unmigratable_content-20210325-28817-9cyo58 and see if the rpms listed in there are important to you, or if they seem like old content you don’t need. If thats the case, you can proceed and run:

foreman-rake katello:approve_corrupted_migration_content

and then continue with the process

Antarctic_Guy · April 15, 2021, 6:11pm

I had run into this exact same issue. After I disabled deb content, the prepare was still failing with over 50K bad packages. All of our repositories are configured for on_demand download. The approve_corrupted_migration_content allowed prepare to finish. The switchover failed with the same error el1psis had. After reading another forum post, I don’t remember which one, about an issue with CentOS or RHEL 8 repos I then removed all RHEL 8 repos and content views. I purged out all other old content views and ran all the commands again. This time the switchover completed without error. I had the RHEL 8 repos added and CVs created but were not actually using them yet as the OS is not yet approved for use in our environment.

el1psis · May 7, 2021, 4:19am

Thanks a lot. It was migrated successfully. I have purged out all other old content views, deleted all products. In my opinion this failed migration was from RHEL 8 too.

pabloalcantara · May 10, 2021, 7:57pm

Hi,
I have done every thing (except delete RHEL 8 and CentOS 8 repos.
How Can I see what repo is causing this possible problem?

[root@server]# foreman-rake katello:pulp3_migration --trace
Rubocop not loaded.
** Invoke katello:pulp3_migration (first_time)
** Invoke dynflow:client (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute dynflow:client
** Execute katello:pulp3_migration
Starting task.
2021-05-10 13:51:40 -0300: Content migration starting.
2021-05-10 14:18:32 -0300: Distribution creation 7/34Migration failed, You will want to investigate: https://server/foreman_tasks/tasks/3fc7cff9-3e8d-4e6e-89ac-59c905e05970
rake aborted!
ForemanTasks::TaskError: Task 3fc7cff9-3e8d-4e6e-89ac-59c905e05970: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/40bdb2f8-ca2a-4d36-b37f-384706381109/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in ’
Tasks: TOP => katello:pulp3_migration

caseybea · May 25, 2021, 3:02pm

So, I am new to the whole migration thing and am kind of stuck.

I am getting the error: WARNING: MISSING OR CORRUPTED CONTENT DETECTED

And the file indicated in the migration stats shows me a list of a THOUSAND rpms. The packages are all from from “8” repos (redhat8, or centos8, or oracle linux 8).

I have tried running the orphaned object cleanup. No dice.

The list of actions to perform includes: “Performing a ‘Verify Checksum’ sync under Advanced Sync Options, let it complete, and re-running the migration”, but I have not been able to find that option.

In my situation, I cannot wipe out all of the centos8/redhat8/etc stuff because that’s half of my entire katello infrastructure.

And “Mark currently corrupted or missing content as skipped” makes me nervous, as with a thousand packages, I cannot tell if ALL of them are “old” or not.

Recommendations?

caseybea · May 25, 2021, 4:32pm

Ah, I finally found the advanced sync (with verify content). I am giving that a shot and will reply back.

Justin_Sherrill · May 25, 2021, 4:57pm

@caseybea awesome, let us know how it goes!

@pabloalcantara sorry for the delay, You may try it again and check the output of ‘journalctl -u pulpcore-worker@*’ to see if any tracebacks are printed.

caseybea · May 25, 2021, 7:32pm

Well, zero progress.

first, I redid ALL the syncs, now with “verify” checked under the advanced options.

second, re-executed the orphaned content delete.

third, re-ran the content prepare.

Same error. SAME exactly 1,000 packages listed in the reference file in the migration stats command.

Help?

Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-05-25 14:10:30 -0500: Importing migrated content type rpm: 1080/1459
Some corrupted or missing content found, run 'foreman-maintain content migration-stats' for more information.
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 255
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Justin_Sherrill · May 25, 2021, 9:09pm

I would do a little bit of investigation. Its possible that there were multiple copies of the packages that it claims are missing/corrupt. For example, lets say it complained about ‘kernel-3.10.0-957.12.2.el7.x86_64’ being missing/corrupt, you can run:

find /var/lib/pulp/content/ | grep kernel-3.10.0-957.12.2.el7.x86_64

My guess is that will show at least one copy on disk. You could also go on a test client and verify that you can yum install one of the listed rpms (basically proving that the rpms mentioned aren’t the ones missing/corrupt).

Another thing you could do is go into the foreman-rake console and run:

Katello::Rpm.where(migrated_pulp3_href: nil).where(filename: ‘kernel-3.10.0-957.12.2.el7.x86_64.rpm’).count
Katello::Rpm.where(migrated_pulp3_href: nil).where(filename: ‘kernel-3.10.0-957.12.2.el7.x86_64.rpm’)[0].repositories.map(&:name)
Katello::Rpm.where(migrated_pulp3_href: nil).where(filename: ‘kernel-3.10.0-957.12.2.el7.x86_64.rpm’)[0].repositories.map(&:content_view).map(&:name)

The first line will print how many unmigrated rpms have that exact rpm filename.
The second line will print the repository names the first unmigrated rpm with that filename is in.
The 3rd line will print the content views those repositories are in.

Its very likely that these corrupt/missing rpms are missing from some past event and since you haven’t noticed until this point, they aren’t really in use.

caseybea · May 25, 2021, 9:34pm

I’ve done several picks of RPMS, they all appear to be from Centos8:

irb(main):012:0> Katello::Rpm.where(migrated_pulp3_href: nil).where(filename: 'rsync-3.1.3-6.el8.x86_64.rpm')[0].repositories.map(&:name)
=> ["centos8_base", "centos8_base", "centos8_base", "centos8_base", "centos8_base", "centos8_base"]
irb(main):013:0> Katello::Rpm.where(migrated_pulp3_href: nil).where(filename: 'rsync-3.1.3-6.el8.x86_64.rpm')[0].repositories.map(&:content_view).map(&:name)
=> ["Centos8", "Centos8", "Default Organization View", "Centos8", "Centos8", "Centos8"]
irb(main):014:0> Katello::Rpm.where(migrated_pulp3_href: nil).where(filename: 'rpm-apidocs-4.14.2-25.el8.noarch.rpm')[0].repositories.map(&:content_view).map(&:name)
=> ["Centos8", "Centos8", "Default Organization View", "Centos8", "Centos8", "Centos8"]

And checking a random sampling of more, the versions do appear to be “older”.

I may first try just wiping all traces of centos8 (deleting the hosts, content views, repos, all of it) and see if that does it.

Any of the rpms listed appear in a TON of places (example below), but the katello command above SEEM to only point to the Centos8 content view.

[root@katello unmigratable_content-20210525-18215-1ivjglw]# locate rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/content/units/rpm/81/c2108792e7185de0bd2e19f50bc768eaa737b451d4c279ac30ad7de8fa9c16/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/content/units/rpm/f5/d24a109074170f865b8cfef767a3b6a1d0497c2965d8442099f824eba9be68/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Centos8-Development-0d69ed0f-5f4e-4058-a99f-e8411e38bf82/1620657772.86/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Centos8-Library-0d69ed0f-5f4e-4058-a99f-e8411e38bf82/1620657418.23/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Centos8-Production-0d69ed0f-5f4e-4058-a99f-e8411e38bf82/1620657790.12/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Centos8-v19_0-0d69ed0f-5f4e-4058-a99f-e8411e38bf82/1607441046.98/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Centos8-v20_0-0d69ed0f-5f4e-4058-a99f-e8411e38bf82/1620657415.0/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Oracle8-Development-57f9d878-d7c7-4910-a157-fb9ddd6d5659/1620658270.27/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Oracle8-Library-57f9d878-d7c7-4910-a157-fb9ddd6d5659/1620657879.37/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Oracle8-Production-57f9d878-d7c7-4910-a157-fb9ddd6d5659/1620658290.45/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Oracle8-v11_0-57f9d878-d7c7-4910-a157-fb9ddd6d5659/1617647575.54/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-Oracle8-v12_0-57f9d878-d7c7-4910-a157-fb9ddd6d5659/1620657869.33/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-RedHat8-Development-40e9c880-d53e-4848-941f-20a77265b849/1620659168.87/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-RedHat8-Library-40e9c880-d53e-4848-941f-20a77265b849/1620658647.0/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-RedHat8-Production-40e9c880-d53e-4848-941f-20a77265b849/1620659185.1/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-RedHat8-v14_0-40e9c880-d53e-4848-941f-20a77265b849/1617648404.04/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-RedHat8-v15_0-40e9c880-d53e-4848-941f-20a77265b849/1620658640.66/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/lib/pulp/published/yum/master/yum_distributor/40e9c880-d53e-4848-941f-20a77265b849/1621426079.95/Packages/r/rsync-3.1.3-6.el8.x86_64.rpm
/var/www/html/centos8/BaseOS/x86_64/kickstart/Packages/rsync-3.1.3-6.el8.x86_64.rpm
/var/www/html/centos8/BaseOS/x86_64/os/Packages/rsync-3.1.3-6.el8.x86_64.rpm