Pulp2 to pulp3 migration doesn't work

I have successfully upgrade katello from 3.13 to 3.18.2, but I can’t migrate pulp2 to pulp3. We also have ubuntu repository, but I have remove it from foreman, and run:

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false

search for debian packages show this in console:

foreman-rake console

Rubocop not loaded.

Loading production environment (Rails 6.0.3.4)

irb(main):001:0> Katello::Deb.count

=> 82045

irb(main):002:0> Katello::Repository.where(:root => Katello::RootRepository.where(:content_type => “deb”)).count

=> 0

I’m getting this error at step 2:

foreman-maintain content prepare

Running Prepare content for Pulp 3

================================================================================

Prepare content for Pulp 3:

Rubocop not loaded.

Starting task.

2021-05-10 13:24:02 +0200: Distribution creation 0/342Migration failed, You will want to investigate: https://foreman/foreman_tasks/tasks/e81e6880-b94c-4f98-a12b-7aa0fd9f6887

rake aborted!

ForemanTasks::TaskError: Task e81e6880-b94c-4f98-a12b-7aa0fd9f6887: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/7f456060-1734-4179-bac8-72c9946e71d3/.

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>’

/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>’

Tasks: TOP => katello:pulp3_migration

(See full trace by running task with --trace)

                                                                  [FAIL]

Failed executing foreman-rake katello:pulp3_migration, exit status 1


Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

[content-prepare]

Resolve the failed steps and rerun

the command. In case the failures are false positives,

use --whitelist=“content-prepare”

Expected outcome:
Pulp2 should upgrade to Pulp 3.

Investigating tasks show this:
Exception:

Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/7f456060-1734-4179-bac8-72c9946e71d3/.

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors’

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each’

/var/log/messages:
May 10 15:47:44 foreman01 journal: cr_repomd_record_fill: File icons.tar.gz doesn’t exists

May 10 15:47:44 foreman01 journal: cr_repomd_record_fill: File icons.tar.gz doesn’t exists

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: pulp: rq.worker:ERROR: Traceback (most recent call last):

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: rv = job.perform()

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: self._result = self._execute()

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: return self.func(*self.args, **self.kwargs)

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 224, in complex_repo_migration

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: migrated_repo.pulp3_repository_version

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 371, in migrate_repo_distributor

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: pulp2dist, repo_version)

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 78, in migrate_to_pulp3

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 320, in publish

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: metadata_signing_service=metadata_signing_service,

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 551, in create_repomd_xml

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: record.fill(checksum_type)

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: OSError: File icons.tar.gz doesn’t exists or not a regular file

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: Traceback (most recent call last):

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: rv = job.perform()

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: self._result = self._execute()

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: return self.func(*self.args, **self.kwargs)

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 224, in complex_repo_migration

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: migrated_repo.pulp3_repository_version

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 371, in migrate_repo_distributor

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: pulp2dist, repo_version)

May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 78, in migrate_to_pulp3

Did I miss something?

I believe the ability to 2to3 migrate deb content in particular will only be available in Katello 3.18.3.
See: Feature #32420: Enable deb content for 2to3 migration - Katello - Foreman

In addition I recommend waiting for the Katello 3.18.4 release (which is planned for quite soon after 3.18.3, and will mainly contain a newer version of the pulp_deb plugin for Pulp 3).

1 Like

OK. What is solution for my case, since I have delete deb repo?

From your /var/log/messages (reposted in block quotes for readability) it looks like some RPM repo is failing to migrate:

May 10 15:47:44 foreman01 journal: cr_repomd_record_fill: File icons.tar.gz doesn’t exists
May 10 15:47:44 foreman01 journal: cr_repomd_record_fill: File icons.tar.gz doesn’t exists
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: pulp: rq.worker:ERROR: Traceback (most recent call last):
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: rv = job.perform()
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: self._result = self._execute()
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: return self.func(*self.args, **self.kwargs)
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 224, in complex_repo_migration
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: migrated_repo.pulp3_repository_version
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 371, in migrate_repo_distributor
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: pulp2dist, repo_version)
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 78, in migrate_to_pulp3
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 320, in publish
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: metadata_signing_service=metadata_signing_service,
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 551, in create_repomd_xml
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: record.fill(checksum_type)
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: OSError: File icons.tar.gz doesn’t exists or not a regular file
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: Traceback (most recent call last):
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: rv = job.perform()
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: self._result = self._execute()
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: return self.func(*self.args, **self.kwargs)
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 224, in complex_repo_migration
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: migrated_repo.pulp3_repository_version
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 371, in migrate_repo_distributor
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: pulp2dist, repo_version)
May 10 15:47:44 cvrtforeman01 pulpcore-worker-1: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 78, in migrate_to_pulp3

@Justin_Sherrill Do you have any ideas (or know who might)?

Can I check which rpm repo have a problem to import into pulp3? If I would know, then I can remove problematic repo.