Katello 3.18 > Debian packages are not fully synchronized on smart proxy

Support
,
1

Katello 3.18 debian packages are not fully synchronized on smart proxy

Problem:

Katello 3.18 debian packages are not fully synchronized on proxies. the all (architecture) packages are missings.

Expected outcome:

Debian packages on smart proxy reflects Debian packages on Katello master

Foreman and Proxy versions:

candlepin-3.1.22-1.el7.noarch
candlepin-selinux-3.1.22-1.el7.noarch
foreman-2.3.5-1.el7.noarch
foreman-cli-2.3.5-1.el7.noarch
foreman-debug-2.3.5-1.el7.noarch
foreman-dynflow-sidekiq-2.3.5-1.el7.noarch
foreman-installer-2.3.5-1.el7.noarch
foreman-installer-katello-2.3.5-1.el7.noarch
foreman-postgresql-2.3.5-1.el7.noarch
foreman-proxy-2.3.5-1.el7.noarch
foreman-release-2.3.5-1.el7.noarch
foreman-selinux-2.3.5-1.el7.noarch
foreman-service-2.3.5-1.el7.noarch
foreman-vmware-2.3.5-1.el7.noarch
katello-3.18.2-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.18.2-1.el7.noarch
katello-debug-3.18.2-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-3.18.2-1.el7.noarch
katello-selinux-3.5.0-1.el7.noarch
katello-server-ca-1.0-3.noarch
puinfktlcmg001.boursorama.fr-foreman-client-1.0-1.noarch
puinfktlcmg001.boursorama.fr-foreman-proxy-1.0-3.noarch
puinfktlcmg001.boursorama.fr-foreman-proxy-client-1.0-1.noarch
puinfktlcmg001.boursorama.fr-qpid-broker-1.0-1.noarch
puinfktlcmg001.boursorama.fr-qpid-client-cert-1.0-1.noarch
puinfktlcmg001.boursorama.fr-qpid-router-client-1.0-1.noarch
puinfktlcmg001.boursorama.fr-qpid-router-server-1.0-1.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.9-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
pulp-selinux-2.21.5-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
pulpcore-selinux-1.2.3-2.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-common-2.21.5-1.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python-pulp-docker-common-3.2.9-1.el7.noarch
python-pulp-oid_validation-2.21.5-1.el7.noarch
python-pulp-puppet-common-2.21.5-1.el7.noarch
python-pulp-repoauth-2.21.5-1.el7.noarch
python-pulp-rpm-common-2.21.5-1.el7.noarch
python-pulp-streamer-2.21.5-1.el7.noarch
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-proton-0.34.0-1.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.11.1-1.el7.noarch
python3-pulp-certguard-1.0.3-1.el7.noarch
python3-pulp-container-2.1.2-1.el7.noarch
python3-pulp-deb-2.9.2-1.el7.noarch
python3-pulp-file-1.3.0-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
python3-pulpcore-3.7.6-1.el7.noarch
python36-qpid-proton-0.34.0-1.el7.x86_64
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.16.0-1.el7.x86_64
qpid-proton-c-0.34.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.7.10-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.4-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.4-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.4-1.el7.noarch
tfm-rubygem-actionview-6.0.3.4-1.el7.noarch
tfm-rubygem-activejob-6.0.3.4-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.4-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.4-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.3.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.8-1.fm2_3.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.17.3-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-vsphere-3.4.0-1.el7.noarch
tfm-rubygem-foreman-tasks-3.0.5-1.fm2_3.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-4.2.2-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.3.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.3.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-0.24.3-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-6.1.2-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.18.2.1-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.3.0-1.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-2.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.4-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-optimist-3.0.0-2.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.4.2-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.1.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.7.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.3.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.9.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch
tfm-rubygem-puma-4.3.6-1.el7.x86_64
tfm-rubygem-puma-plugin-systemd-0.1.5-1.el7.noarch
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.4-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.4-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-rbvmomi-2.2.0-3.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redfish_client-0.5.2-1.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-19.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.6-1.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-1.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.3.0-2.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.3.2-1.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.1-1.fm2_3.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-runtime-6.1-4.el7.x86_64

Foreman and Proxy plugin versions:

foreman-tasks	3.0.5
foreman_remote_execution	4.2.2
katello	3.18.2.1

Foreman Smart Proxy versions:

foreman-proxy-2.3.5-1.el7.noarch

Distribution and version:

Red Hat Enterprise Linux Server release 7.9 (Maipo) (Katello & smart proxies)

Other relevant data:

Katello master has been patched with :

Smart proxies are from scratch, I only added this file (for pulp2):

cat /etc/pulp/server/plugins.conf.d/deb_distributor.json
{
  "publish_default_release": true
}
  • I try to sync Debian buster repositories amd64.

Context

  • I’m abble to subscribe my Debian hosts to Katello itself, works perfectly. (It looks like a problem during smart proxy synchronisation)
  • Smart proxy download policy is immediate.
  • I’m able to subscribe my Redhat hosts to Katello itself and all smart proxies as weel, no packages missing.

Hi,

when trying to subscribe my Debian hosts to a smart proxy I notice all lot of missing packages.

I realize that when synchonizing Debian packages, only packages reflecting the repository architecture (on Katello) are synchronized (amd64 in my case). I mean the *all packages are not synced (and not published) on smart proxies.

When I check how much packages are presents on a smart proxy after a successful synchronization :

find /var/lib/pulp/content/units/deb -type f -name "*.deb" | wc -l
29290

while my content view contains 59375 Debian packages.

When I exclude amd64 packages, I can see that all packages are missing.

find /var/lib/pulp/content/units/deb -type f -name "*.deb" | grep -v amd64 | wc -l
0

I try to find my problem by reading the pulp-deb-plugins-1.10.2-1.el7.noarch code.

I noticed that the case of all architecture package is managed in /usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py:

# The units/packages for arch all need to be appended to
# every other arch:
all_units = arch_units.pop('all', [])
for arch in arch_units:
    arch_units[arch].extend(all_units)
# ...and then be readded to the list of architectures:
arch_units['all'] = all_units

But not in /usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py. So all packages should be published correctly but they aren’t because not synced.

So I try to fix it on the smart proxy with the following (./sync.py is the original file):

cat patch.txt

--- sync.py     2021-06-16 10:55:10.861512486 +0200
+++ /usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py 2021-06-16 14:23:09.997646846 +0200
@@ -231,6 +231,8 @@
         distributions = self.parent.distributions
         components = self.parent.components
         architectures = self.parent.architectures
+        if not "all" in architectures:
+            architectures = architectures.append("all")
         dl_reqs = []
         for distribution in distributions:
             self.verify_release_file(distribution)

Then I restart foreman proxy services, reset data with installer and resync my capsule (full sync).

Now almost all the Debian packages are presents on the smart proxy: 59375 in my CV and 58418.

I can find the all packages in /var/lib/pulp/content/units/deb/ as well the publication (links created) in /var/lib/pulp/published/deb/, even if some oh them are still missing (I’ll try to find which ones).

My fix is just a test but I can confirm that something is missing during the debian repos synchronization on a smart proxy with Katello 3.18.

Perhaps I met this behavior because my customs changes (see above Other relevant data:). But without them, I can’t use debian repos with Katello 3.18.

I would like to share it to know if anybody who use Katello 3.18 with smart proxy to manage Debian 10 repositories already meet this behavior, any feedback are welcomes.

2

actually I have noticed before in the sync code that it may not be synchronizing arch=all package indices correctly in Pulp 3. I have a hand written note on my desk to investigate further… If so, then this is a serious bug on the pulp_deb for Pulp 3 side. I imagine this was not noticed earlier since not all APT repos publish their arch=all packages in separate index (but pulp_deb for Pulp 3 does, hence the issue with the smart proxy sync). I will open an issue with pulp_deb, and try to prioritize it soon.

3

@quba42 Thank you for these explanations. I await your feedback on this bug. Do not hesitate to ask me to test a possible fix on my platform

4

I think I mixed myself up somewhat. The smart proxy performing the sync is using Pulp 2.
In that case it looks like both Pulp 2 and Pulp 3 have trouble synchronizing separate arch=all indices (I was entirely focused on the Pulp 3 case which won’t help your Pulp 2 smart proxy).

5

Yes I confirm you, my problem occurs only with smart proxy (pulp2) not on katello itself (pulp3) where I can subscribe and find all my packages (amd64 & all).

I’m not sure but it seems that pulp3 uses python3-pulp-deb-2.9.2-1.el7.noarch while pulp2 uses pulp-deb-plugins-1.10.2-1.el7.noarch

6

I think I’m starting to understand.

pulp-deb-plugins is provided by Index of /pulp/pulp/stable/latest/7/x86_64
in Katello repos for Katello 3.

Not for Katello 4 because this repo is no longer used. The new package is python3-pulp-deb and is hosted in pulpcore repository.

It’s the same project: GitHub - pulp/pulp_deb: Debian repository plugin for Pulp (pulpproject.org), but different tags and completly rewrite.

The pulp-deb-plugins is installed on Katello host but I’m wondering if it’s not just for pulp2 to pulp3 migration compatiblity, it’s no longer used with a fresh Katello 3.18 installation.

Is it possible that it’s a Katello 3.18 regression because a mix of pulp3 on Katello and pulp2 on smart proxies ?

7

Everything you have said sounds correct to me.

Even though Pulp 3 is a complete rewrite that does not share a single line of code with Pulp 2, both the Pulp 2 (2-master branch) and the Pulp 3 (main branch) version of the pulp_deb plugin is contained in the same git repo. The Pulp 3 version is packaged in the pulpcore-packaging repo and published to here. The package is named python3-pulp-deb. The old Pulp 2 version is published in the repo you linked. Katello 3.18 ships both versions of Pulp so people can perform 2to3 migrations, but it only ever uses one of the two versions at any given time. Switching is a manual action. Katello 4.0 drops everything Pulp 2 related.

Yes. The Pulp 3 version of pulp_deb publishes repos with separate arch=all index files, while Pulp 2 did not do this. However the Pulp 2 version of pulp_deb contains a bug and fails to synchronize the arch=all index files, thus “loosing” those packages during the sync to the smart proxy. A smart proxy running Pulp 3 should not have this problem, (at least if architecture filtering is not being used. With architecture filtering enabled, I have found a similar bug in the Pulp 3 version of pulp_deb as well. But I don’t think smart proxy syncs use architecture filtering)…

8

when I read the pulp_deb code for Katello 4, it seems we have the same problem : all is managed during the publication taks: pulp_deb/publishing.py at main · pulp/pulp_deb · GitHub

But I see nothing like in the synchronization task. So the all packages should not be correctly download on the smart proxy since those packages are offered by Katello as differents architectures (different indexes).

However, it works for Debian hosts who directly consume Katello repositories (at leat for 3.18) via subscription manager.

I wonder if we haven’t made an amalgamation on the notion of all. From a package (deb) perspective, all is the equivalent of noarch for RPM AND all to designate all the architectures when configuring the debian repositories. Which does not have the same meaning as all when we talk about packages that are not linked to a specific architecture.

What I mean is that I don’t see a reason to have separated the packages named all from their original repositories (different indexes).

Thanks again for your support @quba42

9

The reason for the separate arch=all index is that this is the preferred format for current Debian repos (and it is the format used by official Debian repos, for example here). The reason why it is preferred, is because it cuts down on duplicate package index entries. In the old pulp2 way of doing things the arch=all packages were written to every available architecture specific package index, so if you had a repository with more than one architecture, this could lead to significant duplication of metadata.

The reason why you do not see any special handling for arch=all type packages in the pulp3_deb sync code, is because repositories that have the separate all indexes will place the keyword all in the Architectures field in their Release files. Since the pulp3_deb sync code handles all the architectures from the Release file’s Architectures field, this will automatically include arch=all indexes.

In other words: The pulp3_deb sync will in general sync both the arch=all packages as well as the separate indices.

There is currently still a bug when architecture filtering is used. If users specify a subset of the available architectures for a sync. E.g.: Architectures: amd64 in the Katello UI, then that will currently kick out the all architecture from The Release file’s Architectures field. I am working on fixing this bug right now.

11

Thanks for all @quba42.

12

just a quick update to correct my fix mentioned above. My apologies, I pasted a note instead of the fix actually deployed (which doesn’t work because it returns null :wink: ).

Here my fix:

 diff sync.py /usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py
233a234,235
>         if not "all" in architectures:
>             architectures.append("all")

Now all my all debian packages are synced and published on the smart proxies (pulp2). And my debian hosts are able to subscribe and install packages from smart proxies.

Notes Do not use this fix on production please, it’s only for my own tests. I have no idea of ​​its reliability or possible side effects. Just to share.