Sync apt.puppet.com repository failure

Problem:
Trying to sync various DEB packages from apt.puppet.com.
Successfully working when only having bookworm and bullseye, failing when adding jammy to the list of releases/distributions.
Expected outcome:
Sync successfully
Foreman and Proxy versions:
Foreman 3.9.1-1.el8
Foreman and Proxy plugin versions:

Distribution and version:
RHEL 8.9
Other relevant data:
Im not sure how to enable debug level logging for pulp, below is the error text and output log from web GUI.
My thought was to create two repositories under the Puppet Agent product, which would contain all deb and yum packages from the apt.puppet and yum.puppet.com repos…

Errors:

Cannot create repository version since there are newly added packages with the same name, version, and architecture, but a different checksum. If the log level is DEBUG, you can find a list of affected packages in the Pulp log.

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/018e5714-63cc-7b87-9a64-36a8167922ad/",
    "pulp_created"=>"2024-03-19T14:18:57.869+00:00",
    "state"=>"failed",
    "name"=>"pulp_deb.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"9a995b1f-050b-449e-ad6d-f664b77fb731",
    "created_by"=>"/pulp/api/v3/users/1/",
    "started_at"=>"2024-03-19T14:19:07.529+00:00",
    "finished_at"=>"2024-03-19T14:19:19.319+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.11/site-packages/pulpcore/tasking/tasks.py\", line 61, in _execute_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "             ^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.11/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 183, in synchronize\n" +
       "    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n" +
       "  File \"/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 155, in create\n" +
       "    with self.repository.new_version() as new_version:\n" +
       "  File \"/usr/lib/python3.11/site-packages/pulpcore/app/models/repository.py\", line 1105, in __exit__\n" +
       "    repository.finalize_new_version(self)\n" +
       "  File \"/usr/lib/python3.11/site-packages/pulp_deb/app/models/repository.py\", line 99, in finalize_new_version\n" +
       "    handle_duplicate_packages(new_version)\n" +
       "  File \"/usr/lib/python3.11/site-packages/pulp_deb/app/models/repository.py\", line 165, in handle_duplicate_packages\n" +
       "    raise ValueError(message)\n",
      "description"=>
       "Cannot create repository version since there are newly added packages with the same name, version, and architecture, but a different checksum. If the log level is DEBUG, you can find a list of affected packages in the Pulp log."},
    "worker"=>"/pulp/api/v3/workers/018e56f5-1df9-7ff4-9b3d-811c9369cfe7/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"completed",
       "done"=>9},
      {"message"=>"Update ReleaseFile units",
       "code"=>"update.release_file",
       "state"=>"completed",
       "done"=>3},
      {"message"=>"Update PackageIndex units",
       "code"=>"update.packageindex",
       "state"=>"completed",
       "done"=>32},
      {"message"=>"Un-Associating Content",
       "code"=>"unassociating.content",
       "state"=>"completed",
       "done"=>0},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"completed",
       "done"=>360}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/deb/apt/018e5701-8cc0-7910-b648-d987e78217a2/",
      "shared:/pulp/api/v3/remotes/deb/apt/018e5701-88cc-7ef5-af82-dbb05e62d962/",
      "shared:/pulp/api/v3/domains/018e332f-f0d9-798f-8cf8-c0ec3cd38c7b/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>16, "failed"=>1}}

Not sure how your structure looks like. I typically create separate repositories for the different distributions. So Product would be “Puppet”, repository would be one each for the combination of Puppet version and Distribution like “Puppet7-bookworm-amd64” with the URL “http://apt.puppet.com/”, distribution “bookworm”, component “puppet7” and architecture “amd64”. I think this would work here, but I have not tested the specific combination.

Well, i have no structure at the moment. This is a fresh foreman server which will replace our old environment once i get everything tested and running as we want.

My reasoning for just adding one repo with all distros was that there was no mention of any limitation in the foreman docs, the way its written it seems to be the opposite but i can see how it can be interpreted both ways…

Im running into issues with the yum repo as well, even if i just point it at a single distro version, or even foreman repo itself… I will create a separate thread for that.

So, should i not be able to just add all the distros to the same apt repo? I could add all distros and versions separately, but it seems smarter if i can just add it once and i have them all, on demand…

It could work also in this way, but I started with separate repositories based on the structure I have from rpm repositories. The other reason I don’t sync all distributions is that I typically only need Debian and/or LTS versions of Ubuntu and this already takes enough space. So I never tried it in this way.

Ah yeah, i understand.
We have a lot of different distros and versions but i will probably be able to narrow it down a fair bit to be honest. Im just a bit miffed that it didnt work to just add the entire thing.

My thought for the future was to just have it on-demand, and a task to clean it regularly. We dont install puppet agents that often, and very seldom on servers running older distros (unless a new customer shows up and has a dated environment).
We will have a fair bit of new puppet agent installs once we move to the new foreman server tho, as everything will be upgraded from v6 to v8.