Problem:
foreman-maintain advanced procedure run content-prepare
fails with the followin output:
...
2020-12-29 19:08:13 +0100: Migrating rpm content to Pulp 3 rpm 0/172445Migration failed, You will want to investigate: https://sap-repo/foreman_tasks/tasks/4637f84d-20eb-435e-91e8-867000df9daa
rake aborted!
ForemanTasks::TaskError: Task 4637f84d-20eb-435e-91e8-867000df9daa: Katello::Errors::Pulp3Error: The content located in /var/lib/pulp/content/units/rpm/82/f586a3e47b256d881c1cdf4b2afdd02a0ffd815ba59d9324459a659d484be1/evolution-ews-3.28.5-5.el7.i686.rpm is missing or corrupted. Repair it in pulp2 and re-run the migration. Alternatively, run migration with skip_corrupted=True.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/lib/katello/tasks/pulp3_migration.rake:31:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
The error text ot the task:
…
The content located in /var/lib/pulp/content/units/rpm/82/f586a3e47b256d881c1cdf4b2afdd02a0ffd815ba59d9324459a659d484be1/evolution-ews-3.28.5-5.el7.i686.rpm is missing or corrupted. Repair it in pulp2 and re-run the migration. Alternatively, run migration with skip_corrupted=True.
Expected outcome:
Success.
Foreman and Proxy versions:
Systeminformationen
Red Hat Enterprise Linux Server release 7.9 (Maipo)
Linux sap-repo 3.10.0-1160.11.1.el7.x86_64 #1 SMP Mon Nov 30 13:05:31 EST 2020 x86_64 x86_64 x86_64 GNU/Linux
The goal of this plugin is to unify the way of showing task statuses across the Foreman instance. It defines Task model for keeping the information about the tasks and Lock for assigning the tasks to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks.
Ivan Nečas
3.0.2
foreman_remote_execution
Ein Plugin, das Foreman entfernte Ausführung bietet und die Konfig-Verwaltungsfunktion mit einer entfernten Verwaltungsfunktion ergänzt.
Foreman Remote Execution team
4.2.1
katello
Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp.
BTW:
Searching in the mongodb for the file /var/lib/pulp/content/units/rpm/82/f586a3e47b256d881c1cdf4b2afdd02a0ffd815ba59d9324459a659d484be1/evolution-ews-3.28.5-5.el7.i686.rpm
doesn’t find anything.
What’s found is:
…
mongo pulp_database --eval ‘db.units_rpm.find({_storage_path: {$regex: “.evolution-ews-3.28.5-5…”}}, {_storage_path: 1}).pretty()’
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.9
{
“_id” : “25483221-fd49-461f-8eed-a51411f09a0c”,
“_storage_path” : “/var/lib/pulp/content/units/rpm/23/f01c10bce425010108850ac966e9b18615db0845d71bd9b609c355a85e3b46/evolution-ews-3.28.5-5.el8.x86_64.rpm”
}
{
“_id” : “66cc6dbf-9e3c-4cf5-af2d-2a08b30feb30”,
“_storage_path” : “/var/lib/pulp/content/units/rpm/94/68aebab055db3f1761be1a60f361e42b9cfc398a479d332e1f12b0ee230395/evolution-ews-3.28.5-5.el7.i686.rpm”
}
{
“_id” : “677f1e4f-739e-45b7-9ba0-67056198cf32”,
“_storage_path” : “/var/lib/pulp/content/units/rpm/72/05a1f3964bd78522dc321d43f4aef3f11731df3b90408965d02a93acdbc947/evolution-ews-3.28.5-5.el7.x86_64.rpm”
}
…
The following command for all repositories doesn’t fix the inconsistency:
hammer repository synchronize --validate-contents=true --id="…"
I have the same issue. I tried creating the missing folder and RPM, which gets past the error, only to complain about a different RPM. I suspected one of the katello clean rake tasks would solve it, to no avail.
Were you able to fix this? I noticed that the error suggests using skip_corrupted=true. When I dug around the Pulp3 migration documentation, it appears as a parameter. A search of the katello migration ruby files shows migration.rb makes the call to the pulp3 migration API, but provides no way to include skip_corrupted=true. I’m going to try editing the migration.rb to include the skip_corrupted and try again. This is a lab environment, so I’m not worried about destroying my implementation.
This seems to have solved the issue, but also moved on to another error: duplicate key value violates unique constraint “rpm_image_name_path_platforms_dist_188721ef_uniq”
I discovered that Pulp2 uses Postgres, not Mongodb which is used by Pulp3, which explains why we are not seeing these items in mongodb. A quick search of the postgres tables brought up the problem entries, which I’ll be deleting before running the migration again.
Essentially we will provide some output of how many files are corrupt/missing and give you the opportunity to ignore/skip them, or provide more info for correcting them.
Hello
I have updated to new version, which should contain the fix, but I still get this error:
# foreman-maintain content prepare
Running Prepare content for Pulp 3
Prepare content for Pulp 3:
Starting task.
2021-03-29 12:21:16 +0200: Migrating rpm content to Pulp 3 rpm 33832/39122
2021-03-29 12:24:16 +0200: Migrating rpm content to Pulp 3 erratum 0/46346Migration failed, You will want to investigate: https://foreman-test/foreman_tasks/tasks/74f7929c-523a-4df0-a734-6ad7014e5a7e
rake aborted!
ForemanTasks::TaskError: Task 74f7929c-523a-4df0-a734-6ad7014e5a7e: Katello::Errors::Pulp3Error: duplicate key value violates unique constraint “rpm_image_name_path_platforms_dist_188721ef_uniq”
DETAIL: Key (name, path, platforms, distribution_tree_id)=(boot.iso, images/boot.iso, x86_64, 58f1ba5d-ea30-41e4-b8bb-ac3c8d1093a4) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-prepare”
Sorry for the formating, I don’t know how to do it properly.
I have deleted the repository containing boot.iso from Foreman and now I’m getting this error:
# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-03-29 13:51:41 +0200: Importing migrated content type erratum: 44000/46190Migration failed, You will want to investigate: https://foreman-test/foreman_tasks/tasks/a64cbb90-00fe-4c90-a72c-6622030cfcde
rake aborted!
ForemanTasks::TaskError: Task a64cbb90-00fe-4c90-a72c-6622030cfcde: ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column katello_debs.migrated_pulp3_href does not exist
LINE 1: SELECT COUNT(*) FROM "katello_debs" WHERE "katello_debs"."mi...
^
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
I have installed the package pulp-deb-plugins.noarch if it’s what you mean. In the past I tried to synchronize some Ubuntu repository, but now it’s all deleted from Foreman. Is it possible there are some remnants in pulp? How could I detect/clean it?
Right now we don’t support the migration if debian is installed (because the debian plugin doesn’t support migration yet). That is still coming to 3.18.
