Forman 2.3.5 Katello 3.18.4 pulp3 migration deb

Support
1

Still having issues with migrating deb repositories to enable moving to foreman 2.4

Katello 3.18.4 fixed the earlier migration issues but now it is failing at Distribution creation as below

2021-07-23 11:47:12 +1000: Pre-migrating Pulp 2 deb_component content (general i 2021-07-23 11:47:22 +1000: Pre-migrating Pulp 2 deb_component content (general i 2021-07-23 11:47:32 +1000: Pre-migrating Pulp 2 deb_component content (general i 2021-07-23 11:47:42 +1000: Pre-migrating Pulp 2 deb_component content (general i 2021-07-23 11:47:52 +1000: Pre-migrating Pulp 2 deb_component content (general i 2021-07-23 11:48:02 +1000: Pre-migrating Pulp 2 deb_component content (general i 2021-07-23 11:48:13 +1000: Pre-migrating Pulp 2 deb_component content (general i 2021-07-23 11:52:53 +1000: Distribution creation 8/12Migration failed, You will want to investigate: https://xxx.xxx.xxx.xxxt/foreman_tasks/tasks/f969b23e-d627-41ed-ae9f-10c23b70c82a
rake aborted!
ForemanTasks::TaskError: Task f969b23e-d627-41ed-ae9f-10c23b70c82a: Katello::Errors::Pulp3Error: 3 subtask(s) failed for task group /pulp/api/v3/task-groups/e814ca0b-4dff-4e53-b7c3-22849f38ac57/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1

Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

[content-prepare]

Expected outcome:
foreman-maintain content-prepare completed

Foreman and Proxy versions:
foreman-2.3.5-1.el7.noarch
katello-3.18.4-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python3-pulp-certguard-1.0.3-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
python3-pulp-2to3-migration-0.11.1-1.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.11.0-1.el7.noarch
python-pulp-oid_validation-2.21.5-1.el7.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
python-pulp-puppet-common-2.21.5-1.el7.noarch
python-pulp-rpm-common-2.21.5-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
python3-pulp-deb-2.9.2-1.el7.noarch
pulpcore-selinux-1.2.3-2.el7.x86_64
python3-pulp-file-1.3.0-1.el7.noarch
tfm-rubygem-pulp_container_client-2.1.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.3.0-1.el7.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
python-pulp-repoauth-2.21.5-1.el7.noarch
python-pulp-streamer-2.21.5-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-0.1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

2

With this (and all 2to3 migration issues), we really need the full back trace and error from the underlying Pulp task that failed. I don’t know off the top of my head how best to get at that, (other than monitoring your system log during a migration attempt). I can ask around if someone else knows.

@m-bucher Do you happen to know?

3

Also, there is one known bugfix in the 2to3 migration plugin that has not yet been backported to the version used by Katello: Issue #8928: Deb migration fails if importer config contains no Releases - Migration Plugin - Pulp The backport request is here: Backport #8945: Backport 8928 to 0.11 - Migration Plugin - Pulp

If that is the bug you are experiencing there are one or two workarounds I could suggest.

4

All the deb repositories have Releases set

5

In that case I definitely need the full backtrace from the underlying pulp task in order to analyze this any further.

See the following post for an example of what I need: Katello::Errors::Pulp3Error: column deb_aptremote.ignore_missing_package_indices does not exist - #10 by senetm

6

I’ve run it again and added logs

Run
https://pastebin.com/embed_js/Jnm5zfdm

foreman task trace

https://pastebin.com/embed_js/01nQdW2w

pulpcore-worker tracing

https://pastebin.com/embed_js/qs5gkhJ3

7

It looks like there are three failed tasks in all, and I can see three errors in the pulpcore-worker tracing you provided. The last of the three seems to be the same error reported here: Katello::Errors::Pulp3Error: column deb_aptremote.ignore_missing_package_indices does not exist - #3 by senetm For which I have opened a pulp_deb issue here: Issue #9165: KeyError: 'amd64' when creating publications during 2to3 migration - Debian Support - Pulp

The other two errors seem to be a completely different issue.

8

I can’t see any progress on this issue, is anything likely to happen or do I delete all my deb repos upgrade and move on, I can’t wait much longer as there is a puppet issue with sha on this release on rpms on pulp3. fixed in katello4

9

I can’t promise there will be any movement soon. I will get to it when I will get to it, and then there are often quite long release cycles until a change to the migration plugin makes it back into the package used by a given Katello release. :frowning: