[ContentMigration] Error migrating contents to pulp with some deb repositories

Support
1

Problem:

I’d like to upgrade katello from 3.18.4 to latest, but i’m stuck with content migration. Running foreman-maintain content prepare the procedures stop with this message:

[root@foreman01 ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Checking for valid Katello configuraton.
Starting task.
2021-08-31 08:26:39 +0200: Distribution creation 3/5Migration failed, You will want to investigate: https://foreman01/foreman_tasks/tasks/018d5938-4ed3-45ea-b1e5-132d440e3a4c
rake aborted!
ForemanTasks::TaskError: Task 018d5938-4ed3-45ea-b1e5-132d440e3a4c: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/4553c59b-4c88-49f5-8af3-6d78e6d9910b/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Looking at logs i find this:

ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: pulp: rq.worker:ERROR: Traceback (most recent call last):
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: rv = job.perform()
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: self._result = self._execute()
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: return self.func(*self.args, **self.kwargs)
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: migrated_repo.pulp3_repository_version
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: pulp2dist, repo_version)
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/repository.py", line 75, in migrate_to_pulp3
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: publish(repo_version.pk, simple=True, structured=True)
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 156, in publish
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: prc.package
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 195, in add_package
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: self.package_index_files[package.architecture][0]
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: KeyError: 'amd64'
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: Traceback (most recent call last):
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: rv = job.perform()
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: self._result = self._execute()
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: return self.func(*self.args, **self.kwargs)
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: migrated_repo.pulp3_repository_version
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: pulp2dist, repo_version)
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/repository.py", line 75, in migrate_to_pulp3
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: publish(repo_version.pk, simple=True, structured=True)
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 156, in publish
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: prc.package
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 195, in add_package
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: self.package_index_files[package.architecture][0]
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: KeyError: 'amd64'
ago 31 08:16:48 foreman01 pulpcore-worker-2[40724]: pulp: rq.worker:INFO: 40724@foreman01: 2c4cf469-bc97-4a66-a1ae-a4245ae93fa4
ago 31 08:16:49 foreman01 pulpcore-worker-2[40724]: pulp: rq.worker:INFO: 40724@foreman01: Job OK (2c4cf469-bc97-4a66-a1ae-a4245ae93fa4)

Expected outcome:

Content to be migrated without error

Foreman and Proxy versions:

candlepin-3.1.22-1.el7.noarch
candlepin-selinux-3.1.22-1.el7.noarch
foreman-2.3.5-1.el7.noarch
foreman-cli-2.3.5-1.el7.noarch
foreman-debug-2.3.5-1.el7.noarch
foreman-dynflow-sidekiq-2.3.5-1.el7.noarch
foreman-ec2-2.3.5-1.el7.noarch
foreman-installer-2.3.5-1.el7.noarch
foreman-installer-katello-2.3.5-1.el7.noarch
foreman-postgresql-2.3.5-1.el7.noarch
foreman-proxy-2.3.5-1.el7.noarch
foreman-release-2.3.5-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-2.3.5-1.el7.noarch
foreman-service-2.3.5-1.el7.noarch
foreman-vmware-2.3.5-1.el7.noarch
foreman01-foreman-client-1.0-1.noarch
foreman01-foreman-proxy-1.0-1.noarch
foreman01-foreman-proxy-client-1.0-1.noarch
foreman01-qpid-broker-1.0-2.noarch
foreman01-qpid-client-cert-1.0-1.noarch
foreman01-qpid-router-client-1.0-1.noarch
foreman01-qpid-router-server-1.0-1.noarch
katello-3.18.4-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.18.4-1.el7.noarch
katello-debug-3.18.4-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-3.18.4-1.el7.noarch
katello-selinux-3.5.0.1-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.14.1-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.9-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
pulp-selinux-2.21.5-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
pulpcore-selinux-1.2.3-2.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python-pulp-common-2.21.5-1.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python-pulp-docker-common-3.2.9-1.el7.noarch
python-pulp-oid_validation-2.21.5-1.el7.noarch
python-pulp-puppet-common-2.21.5-1.el7.noarch
python-pulp-repoauth-2.21.5-1.el7.noarch
python-pulp-rpm-common-2.21.5-1.el7.noarch
python-pulp-streamer-2.21.5-1.el7.noarch
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-proton-0.34.0-2.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.11.3-1.el7.noarch
python3-pulp-certguard-1.0.3-1.el7.noarch
python3-pulp-container-2.1.2-1.el7.noarch
python3-pulp-deb-2.9.2-1.el7.noarch
python3-pulp-file-1.3.0-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
python3-pulpcore-3.7.6-1.el7.noarch
python36-qpid-proton-0.34.0-2.el7.x86_64
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.16.0-1.el7.x86_64
qpid-proton-c-0.34.0-2.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.7.10-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.4-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.4-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.4-1.el7.noarch
tfm-rubygem-actionview-6.0.3.4-1.el7.noarch
tfm-rubygem-activejob-6.0.3.4-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.4-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.4-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.3.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-arel-9.0.0-4.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-awesome_print-1.8.0-5.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-coffee-rails-5.0.0-1.el7.noarch
tfm-rubygem-coffee-script-2.4.1-4.el7.noarch
tfm-rubygem-coffee-script-source-1.12.2-4.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-declarative-0.0.10-2.el7.noarch
tfm-rubygem-declarative-option-0.1.0-2.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.8-1.fm2_3.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-ethon-0.12.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-execjs-2.7.0-4.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.17.3-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-aws-3.6.5-1.el7.noarch
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-digitalocean-0.4.0-2.el7.noarch
tfm-rubygem-fog-google-1.11.0-1.el7.noarch
tfm-rubygem-fog-json-1.2.0-3.el7.noarch
tfm-rubygem-fog-openstack-1.0.8-3.el7.noarch
tfm-rubygem-fog-ovirt-1.2.5-1.el7.noarch
tfm-rubygem-fog-vsphere-3.4.0-1.el7.noarch
tfm-rubygem-fog-xenserver-1.0.0-2.el7.noarch
tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
tfm-rubygem-foreman-tasks-3.0.6-1.fm2_3.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible-6.1.1-1.fm2_3.el7.noarch
tfm-rubygem-foreman_ansible_core-4.0.0-1.fm2_3.el7.noarch
tfm-rubygem-foreman_docker-5.0.0-3.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-4.2.2-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-foreman_virt_who_configure-0.5.5-1.fm2_3.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-google-api-client-0.33.2-1.el7.noarch
tfm-rubygem-google-cloud-env-1.3.3-1.el7.noarch
tfm-rubygem-googleauth-0.13.1-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.3.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.3.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.1-1.fm2_3.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-0.24.4-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-httpclient-2.8.3-3.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-6.1.2-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.18.4-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.3.0-1.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-memoist-0.16.0-2.el7.noarch
tfm-rubygem-method_source-0.9.2-2.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.4-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-optimist-3.0.0-2.el7.noarch
tfm-rubygem-os-1.0.0-2.el7.noarch
tfm-rubygem-ovirt-engine-sdk-4.3.0-1.el7.x86_64
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-passenger-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-10.13.el7.x86_64
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.11.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.4.2-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.1.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.9.1-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
tfm-rubygem-pulp_file_client-1.3.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-0.1.el7.noarch
tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch
tfm-rubygem-puma-4.3.6-1.el7.x86_64
tfm-rubygem-puma-plugin-systemd-0.1.5-1.el7.noarch
tfm-rubygem-qpid_messaging-1.38.0-1.el7.x86_64
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.4-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.4-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rake-compiler-1.0.7-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-rbovirt-0.1.7-4.el7.noarch
tfm-rubygem-rbvmomi-2.2.0-3.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redfish_client-0.5.2-1.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-representable-3.0.4-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-retriable-3.1.2-2.el7.noarch
tfm-rubygem-rkerberos-0.1.5-19.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.6-1.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-1.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-signet-0.14.0-1.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.3.0-2.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.3.2-1.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.1-1.fm2_3.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-trollop-2.1.2-3.el7.noarch
tfm-rubygem-typhoeus-1.3.1-1.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-uber-0.1.0-2.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-wicked-1.3.3-2.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-rubygem-zest-0.0.4-1.el7.noarch
tfm-runtime-6.1-4.el7.x86_64

Foreman and Proxy plugin versions:

Distribution and version:

CentOS 7.9
Katello 3.18.4

Other relevant data:
I did upgrades step by step from 3.14.1 to 3.18.4

2

I have been looking through the code to try and come up with a theory of how this error could come about, and I just had a plausible idea.

Do you know if people ever made use of Katello’s upload package feature for Debian repositories?

See (6) in the following image for what I mean: https://docs.orcharhino.com/or/docs/_images/images/repositories.png (The image is of a RPM repository but the relevant part looks the same for deb repositories.)

I think it might be possible the Migration plugin is not properly considering the architecture of such solo packages during migration, since they are not assigned to any Pulp 2 DebComponent. (I really can’t think of any other explanation right now…)

To confirm, I will need to build a test scenario first. Even if my hunch is correct, it will not be trivial to build a fix. The only simple options I can come up with is to wait for a fix, or to upgrade your installation without migrating the Debian content. You could try manually adding the missing architectures to the affected Pulp 3 repositories, but that would require very detailed Pulp 3 knowledge (if it works at all, I am not aware of ever having tested this) and I have no idea how many affected repositories there are in your Katello instance…

3

Actually I just had an idea for something that might work as a simple workaround. (At least it should confirm if my theory of the issue is correct or not.)

On your machine, can you open the following file:

/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/repository.py

Line 75 should be the following:

             publish(repo_version.pk, simple=True, structured=True)

In this line change structured=True to structured=False.

Once you have saved this patch, restart your pulp instance to make sure the change takes effect (systemctl restart pulp* should do it I think), then re-run foreman-maintain content prepare. If I am right, the migration run should now be successful. (If you then switch your Katello to the Pulp 3 back-end it might still fail to publish new versions of affected repos because Katello also uses structured=True for deb publications.)

1 Like
4

Update: My theory that this might be caused by manually uploaded packages is most likely wrong, nevertheless the workaround I suggested should work (it working just won’t tell me anything about why the architecture in question was missing in the first place).

5

No, i do not. They asked me help for fixing a repository sync issue, and then to solve the problem i decided to upgrade to latest.

6

Ok, i’ll try. But will be pulp3 backend be ok then? Can i use it without breaking anything?

7

It won’t break anything more than it already is, but some repos that were migrated might not publish without adding a workaround to Katello. Best case scenario most repos migrated fine and it is only one repo that has the problem. Worst case scenario many repos are affected. The error you encountered is in the very last stage of the migration so most things should be fully migrated by then. If it is just one repo, the least invasive workaround would be to disable structured publishing for that repo only (or maybe just delete and re-create it)… If there are many affected repos, it would be possible to disable structured publishing globally. Katello features should all work with simple publish only (but lacking the structured repos is not very intuitive for users). Here is a high level description of what “structured” and “simple” publish modes do: Feature Overview — Pulp deb Support 2.15.0 documentation

Short of not migrating deb content (and starting from scratch), or waiting until I have built a fix (and I can’t give a clear time table for when that might be), I don’t see many other options. You could try to buy consulting services from my employer ATIX in which case my time might be re-prioritized to find the best possible solution for your instance as quickly as possible…

1 Like
8

I should also warn you that for Pulp 3 based Katello some important features for deb content do not work yet. The most important one I am aware of is the smart proxy sync for deb content. (A colleague of mine is working on that, but it isn’t done yet.)

9

Ok, maybe is better for me to wait for an improved deb support. Is not critical but at this moment the release i’m running works without issues and i can postpone upgrades to 4.x

10

@remix_tj I have now spent some significant time investigating and testing around the code where you experienced the migration bug, and I have come to the conclusion that the bug you experienced is only possible if some part of the deb content migration pipeline was somehow skipped. The error you got can only happen if some migrated Pulp 3 repository contains a PackageReleaseComponent unit for some migrated package, but does not contain the ReleaseArchitecture unit corresponding to that same package. Since ReleaseArchitecture and PackageReleaseComponent content are literally both created by the same loop during migration, this just should not be possible (if all the code was successfully run, and everything was successfully saved to the DB, and properly associated with the right repo).

If you do want to continue with the migration attempt, I would consider dropping all migration progress from the Pulp 3 instance, and attempting a completely fresh migration run. If that is acceptable on your end, I would also be willing to join you for a video conference session, to try and find the exact repo on your instance that produced this bug (perhaps there is some weird corner case that only makes sense when I see it). I do have a strong interest in not having the deb content migration producing such failures…

11

Hello,

if you can send me how to drop migrations i can start cleaning up the env. I can plan then migration with you in video call starting from next week.

Luca

12

@Justin_Sherrill What was that command for dropping all 2to3 migration progress again? (I should note it down prominently since I keep forgetting and then asking you…)