Debian Repo sync end with the error "This field may not be blank"

Support
1

Problem:

I create a Debian Buster and Bullseye Repo for HashiCorp and get an Error by the sync:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/6cd06263-1630-4e5b-a0f7-4c499ec76f67/",
    "pulp_created"=>"2021-12-20T17:01:34.414+00:00",
    "state"=>"failed",
    "name"=>"pulp_deb.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"c135d6ab-f794-429d-acc7-323ba17b0867",
    "started_at"=>"2021-12-20T17:01:34.599+00:00",
    "finished_at"=>"2021-12-20T17:01:36.734+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 317, in _perform_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 124, in synchronize\n" +
       "    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
       "    loop.run_until_complete(pipeline)\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 448, in run\n" +
       "    *[self._handle_distribution(dist) for dist in self.remote.distributions.split()]\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 513, in _handle_distribution\n" +
       "    release_file.components, self.remote.components, distribution\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 562, in _handle_component\n" +
       "    await asyncio.gather(*pending_tasks)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 619, in _handle_package_index\n" +
       "    serializer.is_valid(raise_exception=True)\n" +
       "  File \"/usr/lib/python3.6/site-packages/rest_framework/serializers.py\", line 228, in is_valid\n" +
       "    raise ValidationError(self.errors)\n",
      "description"=>
       "{'section': [ErrorDetail(string='This field may not be blank.', code='blank')], 'priority': [ErrorDetail(string='This field may not be blank.', code='blank')]}"},
    "worker"=>"/pulp/api/v3/workers/69481263-9338-4489-a9fe-031e29259801/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>3},
      {"message"=>"Update ReleaseFile units",
       "code"=>"update.release_file",
       "state"=>"canceled",
       "done"=>1},
      {"message"=>"Update PackageIndex units",
       "code"=>"update.packageindex",
       "state"=>"canceled",
       "done"=>1},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>5}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/deb/apt/e57d7614-7eaf-4f36-aca7-e38687185bb6/",
      "/pulp/api/v3/remotes/deb/apt/39803006-1476-4e6b-8f2c-71f62c6321e5/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>3, "failed"=>1}}

Expected outcome:

The repo sync run successful .

Foreman and Proxy versions:

Foreman/Proxy 3.0.1

Foreman and Proxy plugin versions:

foreman-tasks 5.1.1
foreman_ansible 6.4.1
foreman_puppet 	1.0.5
foreman_remote_execution 4.8.0
katello 4.2.1

Distribution and version:

Red Hat Enterprise Linux Server release 7.9 (Maipo)

Other relevant data:

Command to create the repo:

wget https://apt.releases.hashicorp.com/gpg -O RPM-GPG-KEY-HASHICORP
sudo hammer content-credentials create --organization 'example.com' \
       --path 'RPM-GPG-KEY-HASHICORP' --name 'RPM-GPG-KEY-HASHICORP' \
       --content-type gpg_key
RPM_GPG_KEY_HASHICORP=$(sudo hammer content-credentials list --organization example.com | grep RPM-GPG-KEY-HASHICORP | awk '{split($0,a); print a[1]}')


sudo hammer product create --name 'HashiCorp' --organization 'example.com' \
       --description 'HashiCorp Repos' --gpg-key-id $RPM_GPG_KEY_HASHICORP \
       --sync-plan 'Weekly_Sync'
sudo hammer product list --organization 'example.com'

sudo hammer repository create --organization 'example.com' --product 'HashiCorp' \
  --name 'Debian Bullseye HashiCorp' --label 'DebianBullseye_HashiCorp' \
  --content-type 'deb' --deb-architectures 'amd64' --deb-components 'main' --deb-releases 'bullseye' \
  --gpg-key-id $RPM_GPG_KEY_HASHICORP \
  --url 'https://apt.releases.hashicorp.com/' \
  --mirror-on-sync 'yes'

sudo hammer repository create --organization 'example.com' --product 'HashiCorp' \
  --name 'Debian Buster HashiCorp' --label 'DebianBuster_HashiCorp' \
  --content-type 'deb' --deb-architectures 'amd64' --deb-components 'main' --deb-releases 'buster' \
  --gpg-key-id $RPM_GPG_KEY_HASHICORP \
  --url 'https://apt.releases.hashicorp.com/' \
  --mirror-on-sync 'yes'
2

Hi @jeschero and welcome to the Foreman Community!

Thanks for reporting this issue. I tried reproducing it and ran into a different issue. The sync works if you set the download policy.

For now, I recommend using the download policy “immediate” as a workaround. Using “on demand” does not work in pulp3 right now, but we’re working on it. This PR adds a default value for download polices to deb repositories.

# hammer repository create --help
[...]
 --download-policy DOWNLOAD_POLICY   Download policy for yum repos (either 'immediate' or 'on_demand'); Possible value(s): 'immediate', 'on_demand'

For more information, see Changing the default download policy in the content management guide.

I will keep you posted once there’s some progress.

Cheers,
Maximilian.

1 Like
3

Hi @maximilian, thanks for your answer but this doesn’t solve my problem because the setting Default Custom Repository download policy was by immediate. I changed it in the past for the deb repos.

Have you another idea?

4

Not yet. I’ve tested this on Foreman 2.5 with Katello 4.1 and pulp_deb 2.13 myself successfully. I will ask my colleagues about it.

5

So have you explicitly tried setting the default download policy explicitly for that repo? There is a default setting as you said, but this default setting is not applied to deb repos as of now. This is fixed in the PR that @maximilian mentioned.

6

Hi @nadjaheitmann, in the version foreman 3.0.1 with katello 4.2.1 can’t i set the Download Policy for a debian repo. This option not exist.

I create a snapshot of my foreman vm and upgraded it to foreman 3.1 with katello 4.3.0.rc4. Now i have the option Download Policy but i get the same error, no matter if the option set to On Demand or Immediate.

The Error:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/f0c4f436-eff3-4ae1-afa5-8e3c7f44239b/",
    "pulp_created"=>"2021-12-22T11:21:31.383+00:00",
    "state"=>"failed",
    "name"=>"pulp_deb.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"c4b74ae4-ee30-4ad2-bd19-f10353dd0bf7",
    "started_at"=>"2021-12-22T11:21:31.671+00:00",
    "finished_at"=>"2021-12-22T11:21:56.701+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x34738 bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x3474c bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x34760 bytes=137 start="  File \"/o" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x34774 bytes=75 start="    DebDec" end=".create()\n"> +
       #<Sequel::SQL::Blob:0x34788 bytes=142 start="  File \"/o" end="in create\n"> +
       #<Sequel::SQL::Blob:0x3479c bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x347b0 bytes=110 start="  File \"/o" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x347c4 bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x347d8 bytes=135 start="  File \"/o" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x347ec bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x34800 bytes=127 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x34814 bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x34828 bytes=129 start="  File \"/o" end="1, in run\n"> +
       #<Sequel::SQL::Blob:0x3483c bytes=26 start="    await " end="o.gather(\n"> +
       #<Sequel::SQL::Blob:0x34850 bytes=146 start="  File \"/o" end="tribution\n"> +
       #<Sequel::SQL::Blob:0x34864 bytes=37 start="    await " end="ub_tasks)\n"> +
       #<Sequel::SQL::Blob:0x34878 bytes=143 start="  File \"/o" end="component\n"> +
       #<Sequel::SQL::Blob:0x3488c bytes=41 start="    await " end="ng_tasks)\n"> +
       #<Sequel::SQL::Blob:0x348a0 bytes=147 start="  File \"/o" end="age_index\n"> +
       #<Sequel::SQL::Blob:0x348b4 bytes=46 start="    serial" end="ion=True)\n"> +
       #<Sequel::SQL::Blob:0x348c8 bytes=128 start="  File \"/o" end=" is_valid\n"> +
       #<Sequel::SQL::Blob:0x348dc bytes=39 start="    raise " end="f.errors)\n">,
      "description"=>
       "{'section': [ErrorDetail(string='This field may not be blank.', code='blank')], 'priority': [ErrorDetail(string='This field may not be blank.', code='blank')]}"},
    "worker"=>"/pulp/api/v3/workers/1c2e6cb9-5268-4846-a7d9-ae76930ce01a/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>3},
      {"message"=>"Update ReleaseFile units",
       "code"=>"update.release_file",
       "state"=>"canceled",
       "done"=>1},
      {"message"=>"Update PackageIndex units",
       "code"=>"update.packageindex",
       "state"=>"canceled",
       "done"=>1},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>4}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/deb/apt/e57d7614-7eaf-4f36-aca7-e38687185bb6/",
      "shared:/pulp/api/v3/remotes/deb/apt/39803006-1476-4e6b-8f2c-71f62c6321e5/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>17, "failed"=>1}}

One more note, I have included other Debian repos and there I have no problem with the sync.

7

@jeschero Thanks for checking and providing more logs. We will try to reproduce the error once again.

8

Hey @jeschero
My colleague @m-bucher had a look at this again with the latest version of pulp_deb.
It turns out the hashicorp repo is rather special as the PackageIndex contains two fields that are defined but empty. As of right now, we assume that this edgecase requires a change in pulp_deb.

Pulp Issue Tracker #9658

9

This has been fixed on master (but not yet released), see: Issue #8770: Package validation fails if package paragraphs contain keys without values - Debian Support - Pulp

10

Thanks for the information @maximilian and @quba42.

@quba42, can you tell me when it will be released?

11

This is actually a somewhat complicated question.
Since there was a new pulpcore release, a regular pulp_deb release is overdue, but that won’t help you, since you really want to know when the fix will be available in Katello.

We don’t currently have any set process for backporting pulp_deb fixes, but this fix is probably a good candidate for backporting. Here is what I will do: I will discuss backporting this fix in our next Pulpcore/katello/pulp_deb integration meeting - HackMD. This is currently scheduled for the 4th of January. I will report back after that.

12

thank you.

Until the time the fix is available have you temporary fix which i can use?

13

You can try manually applying the fix from the upstream Pulp issue: Drop package paragraph fields without values · pulp/pulp_deb@67cb51a · GitHub
However, I have not tested this, and I do not know if there are any conflicts with the pulp_deb version you are using.

14

Thanks for this information, i tested it but don’t work for me.

I modifier the function from822 to this:

org

    @classmethod
    def from822(cls, data, **kwargs):
        """
        Translate deb822.Package to a dictionary for class instatiation.
        """
        return cls(
            data={k: data[v] for k, v in cls.TRANSLATION_DICT.items() if v in data}, **kwargs
        )

new

    @classmethod
    def from822(cls, data, **kwargs):
        """
        Translate deb822.Package to a dictionary for class instatiation.
        """
        skip = ["Filename", "MD5sum", "Size", "SHA1", "SHA256", "SHA512"]
        package_fields = {}
        custom_fields = {}
        for k, v in data.items():
            #if k in cls.TRANSLATION_DICT_INV:
            #    key = cls.TRANSLATION_DICT_INV[k]
            #    package_fields[key] = v
            #elif k not in skip:
                # also save the fields not in TRANSLATION_DICT
                custom_fields[k] = v

        unique_package_name = "{}_{}_{}".format(
            package_fields["package"],
            package_fields["version"],
            package_fields["architecture"],
        )

        # Drop keys with empty values
        empty_fields = [k for k, v in package_fields.items() if not v]
        for key in empty_fields:
            message = _('Dropping empty "{}" field from "{}" package!').format(
                key, unique_package_name
            )
            log.warning(message)
            del package_fields[key]

        # Delete package fields with values of incorrect type
        if "installed_size" in package_fields:
            try:
                int(package_fields["installed_size"])
            except (TypeError, ValueError):
                log.warn(
                    _(
                        "Dropping 'Installed-Size' field from '{}', "
                        "since the value '{}' is of incorrect type."
                    ).format(unique_package_name, package_fields["installed_size"])
                )
                del package_fields["installed_size"]
        message = _(
            "Dropping '{}' field from package '{}', "
            "since the value '{}' is not in the allowed values list '{}'"
        )
        bool_values = [value[1] for value in BOOL_CHOICES]
        if "essential" in package_fields and package_fields["essential"] not in bool_values:
            log.warn(
                message.format(
                    "Essential", unique_package_name, package_fields["essential"], bool_values
                )
            )
            del package_fields["essential"]
        if (
            "build_essential" in package_fields
            and package_fields["build_essential"] not in bool_values
        ):
            log.warn(
                message.format(
                    "Build-Essential",
                    unique_package_name,
                    package_fields["build_essential"],
                    bool_values,
                )
            )
            del package_fields["build_essential"]
        if "multi_arch" in package_fields:
            allowed_values = [value[1] for value in BasePackage.MULTIARCH_CHOICES]
            if package_fields["multi_arch"] not in allowed_values:
                log.warn(
                    message.format(
                        "Multi-Arch",
                        unique_package_name,
                        package_fields["multi_arch"],
                        allowed_values,
                    )
                )
                del package_fields["multi_arch"]

        package_fields["custom_fields"] = custom_fields
        return cls(data=package_fields, **kwargs)

I need to modifier the part in for k, v in data.items(): otherwise i get the error Katello::Errors::Pulp3Error: type object 'Package822Serializer' has no attribute 'TRANSLATION_DICT_INV'.

Now i have an other the problem. The repo sync finished, but i can’t see any package in the repo overview. In the sync process, i can see the packages.

15

@jeschero Ok, the fix you want is now released as part of pulp_deb 2.17.0.

In addition, I have backported it to the pulp_deb 2.16.1 release as well.
I have opened a PR to have the 2.16.1 version packaged as an RPM package here.
Once this PR is merged, and the pipeline is run, pulp_deb 2.16.1 will be available for Katello 4.3 installations. Katello 4.3 is currently on RC4 and expected for a full release any day now.

This is as far as I am willing to backport the fix. This should provide you with an upgrade path to an official release containing the fix within a couple of days. I hope that helps.

16

@quba42 Thank you for your update. After the release of Katello 4.3, i will test it and we can close this issue.

17

Note that while my PR for pulpcore-packaging is now merged, there is not yet a built tfm-pulpcore-python3-pulp-deb-2.16.1-1.el7.noarch.rpm package available in the corresponding RPM repos. Make sure you it is available before you upgrade. If it still isn’t available in a couple of days, ping somebody so they can investigate.

18

Update: The pulp_deb 2.16.1 RPM package is now available in the repo used for Katello 4.3.

19

Thanks for this update.

Katello 4.3 is not yet released (Katello 4.3.0 RC release process), so i need to wait.

20

@jeschero Took longer than expected, but: Katello 4.3 GA is available