Upload deb package through hammer results in empty/invalid packages in repo

Problem:
We want to upload a custom package each day so we can install and upgrade it overtime to know when a system is patched.
I’ve made a simple .deb file that makes a file in json format in /etc/ansible/facts.d/ and if i install it it also works. If i upload it through the UI that also works fully. But it’s when i upload it with hammer that i get an empty row and the subscription-manager clients also report they can’t find the package.

Looking with hammer, i see in the deb-package list an ID created but not FILENAME:


Trying to remove said .deb also resulsts in a task stuck:

Error the clients get once we upload the package using hammer:
Failed to fetch katello://foreman.server.lan/pulp/deb/comp/Production/CV_UBUNTU_22-04/custom/Today_I_Patched_Package/Today_I_Patched_Package/dists/default/all/binary-amd64/Packages 404 Not Found

As root i upload the package from the current directory like so:
hammer repository upload-content --path pocket_mouse.deb --organization comp --product “Today I Patched Package” --name “Today I Patched Package”

A seperate product and repo is made with the same name, so we can include the product into every CV.

The upload also goes smoothly: Successfully uploaded file pocket_mouse.deb

I’ve also tried a package that was sync’d from ubuntu and that one also has the same behavior, so i believe the way i packaged shouldn’t be the issue.

Does anyone have a suggestion how i can find more information how to find a possible workaround/the root cause that i or maybe we can provide to fix in foreman itself (if applicable)?

Expected outcome:
deb package is uploaded if it isn’t already in the repository.

Foreman and Proxy versions:
3.11.2

Foreman and Proxy plugin versions:

Installed Packages

  • ansible-collection-theforeman-foreman-4.0.0-2.el9.noarch
  • candlepin-4.4.14-1.el9.noarch
  • candlepin-selinux-4.4.14-1.el9.noarch
  • dynflow-utils-1.6.3-1.el9.x86_64
  • foreman-3.11.2-1.el9.noarch
  • foreman-cli-3.11.2-1.el9.noarch
  • foreman-dynflow-sidekiq-3.11.2-1.el9.noarch
  • foreman-installer-3.11.2-1.el9.noarch
  • foreman-installer-katello-3.11.2-1.el9.noarch
  • foreman-obsolete-packages-1.9-1.el9.noarch
  • foreman-postgresql-3.11.2-1.el9.noarch
  • foreman-proxy-3.11.2-1.el9.noarch
  • foreman-redis-3.11.2-1.el9.noarch
  • foreman-release-3.11.2-1.el9.noarch
  • foreman-selinux-3.11.2-1.el9.noarch
  • foreman-service-3.11.2-1.el9.noarch
  • katello-4.13.1-1.el9.noarch
  • katello-ca-consumer-vcdsat01.virtualcenter.lan-1.0-7.noarch
  • katello-certs-tools-2.10.0-1.el9.noarch
  • katello-client-bootstrap-1.7.9-2.el9.noarch
  • katello-common-4.13.1-1.el9.noarch
  • katello-repos-4.13.1-1.el9.noarch
  • katello-selinux-5.0.2-1.el9.noarch
  • pulpcore-obsolete-packages-1.2.0-1.el9.noarch
  • pulpcore-selinux-2.0.1-1.el9.x86_64
  • python3.11-pulp-ansible-0.21.8-1.el9.noarch
  • python3.11-pulp-cli-0.27.2-1.el9.noarch
  • python3.11-pulp-container-2.20.3-1.el9.noarch
  • python3.11-pulp-deb-3.2.1-1.el9.noarch
  • python3.11-pulp-glue-0.27.2-1.el9.noarch
  • python3.11-pulp-python-3.11.3-1.el9.noarch
  • python3.11-pulp-rpm-3.26.1-1.el9.noarch
  • python3.11-pulpcore-3.49.19-1.el9.noarch
  • rubygem-dynflow-1.8.4-1.el9.noarch
  • rubygem-foreman-tasks-9.1.1-1.fm3_11.el9.noarch
  • rubygem-foreman_ansible-14.0.0-2.fm3_11.el9.noarch
  • rubygem-foreman_azure_rm-2.3.0-1.fm3_11.el9.noarch
  • rubygem-foreman_discovery-24.0.1-2.fm3_11.el9.noarch
  • rubygem-foreman_maintain-1.6.9-1.el9.noarch
  • rubygem-foreman_openscap-8.0.2-1.fm3_11.el9.noarch
  • rubygem-foreman_remote_execution-13.1.0-1.fm3_11.el9.noarch
  • rubygem-foreman_rh_cloud-9.0.56-1.fm3_11.el9.noarch
  • rubygem-hammer_cli-3.11.0-1.el9.noarch
  • rubygem-hammer_cli_foreman-3.11.0-1.el9.noarch
  • rubygem-hammer_cli_foreman_ansible-0.7.0-1.fm3_11.el9.noarch
  • rubygem-hammer_cli_foreman_azure_rm-0.3.1-1.fm3_11.el9.noarch
  • rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.el9.noarch
  • rubygem-hammer_cli_foreman_tasks-0.0.21-1.fm3_11.el9.noarch
  • rubygem-hammer_cli_foreman_virt_who_configure-0.1.0-1.fm3_10.el9.noarch
  • rubygem-hammer_cli_katello-1.13.0-0.2.pre.master.el9.noarch
  • rubygem-katello-4.13.1-1.el9.noarch
  • rubygem-pulp_ansible_client-0.21.3-1.el9.noarch
  • rubygem-pulp_certguard_client-3.49.6-1.el9.noarch
  • rubygem-pulp_container_client-2.20.0-1.el9.noarch
  • rubygem-pulp_deb_client-3.2.0-1.el9.noarch
  • rubygem-pulp_file_client-3.49.6-1.el9.noarch
  • rubygem-pulp_ostree_client-2.3.0-1.el9.noarch
  • rubygem-pulp_python_client-3.11.1-1.el9.noarch
  • rubygem-pulp_rpm_client-3.26.1-1.el9.noarch
  • rubygem-pulpcore_client-3.49.6-1.el9.noarch
  • rubygem-puppetdb_foreman-6.0.2-1.fm3_10.el9.noarch
  • rubygem-smart_proxy_dynflow-0.9.2-1.fm3_11.el9.noarch
  • rubygem-smart_proxy_pulp-3.3.0-1.el9.noarch

Distribution and version:
RHEL 9.4

Other relevant data:

So I tried reproducing this with a random package (just to make sure there isn’t anything specific to your self built package that is causing this).

My example package was vim_9.0.1378-2_amd64.deb.

In my case, uploading the package sometimes resulted in what you described, depending on what hammer options I used. The following resulted in a blank line package entry:

hammer repository upload-content --path vim_9.0.1378-2_amd64.deb --id 123 --content-type deb

However the following appears to have worked:

hammer repository upload-content --path vim_9.0.1378-2_amd64.deb --id 124 --content-type deb --organization Example

In both cases Hammer claimed success:

[...............................................] [100%]
Successfully uploaded file vim_9.0.1378-2_amd64.deb

I initially tried supplying both --id and --product-id, but Hammer complained I can’t supply both of those. This does not make any sense to me, since supplying just a product does not yet uniquely identify a Katello repository, but o well…

@Macley Can you perhaps try adding the --content-type deb flag and parameter to your command? I recommend testing the upload on an entirely new otherwise empty repository to see if that changes the result.

Even if some combination of Hammer flags end up resulting an a working deb upload, other combinations should not result in a broken state. They should either fail hard with a meaningful error or fully succeed. So that is definitely a bug.

Can you please open an issue at Issues - Foreman?

I will also raise this internally.

1 Like

Ok, I did some more testing and it looks like what I reported in my previous post is not quite right.

It is not the case that this worked depending on what Hammer options I use, but rather it appears to always fail the first time I upload the package in question to a repo. Then If I try to upload that same package again to a different repo, I now appear to get a working state. Re-upload to the first repo that contains the broken state, results in that repo reporting that it now contains 2 packages, one of which is the broken state, and one is what looks like a working state.

Pretty weird, definitely a bug.

1 Like

Hi! Just wanted to try out your first suggestion! But the tests you did futher definitly align with what i’m experiencing aswell! I will make a bug report!
Thank you so much already for responding so quickly and giving such throughout explaination! It’s very appricated!
Btw, i do believe if you do the same with an rpm package that you won’t have this issue (tho i only tested this with Satellite). Will update this message to include the bug report i’ve made!

Please post a link to the issue you create in this thread, so I can find it. :wink:

1 Like

Have to wait for an admin to approve my account :slight_smile: But will write down a message in notepad so i can post it immediatly! Definitly will link to this discussion aswell!

sorry still can’t upload the bug to the site, have to wait still for my account to be approved.

Done.

You should have received a mail with a verification link, tho :slight_smile:

1 Like

thank you very much you two! I have created the bug here:

Sorry for submitting it with having the resolved in version ticked, i tried to untick it but it didn’t let me sadly (and i think it’s less ideal to recreate the whole bug report).

It’s a low prioty issue (IMO) as double uploading does fix it in a way that the package can be downloaded, even when upgraded!

1 Like

Fixes merged into upstream Katello:

3 Likes

Thank you so much! <3