Duplicate key value violates unique constrain when synchronizing product published deb file

Problem:
When i update my two products which are the 18.04 and 20.04 ubuntu repo’s (the default onces). They give the following errors:
18.04
duplicate key value violates unique constraint “core_publishedartifact_publication_id_relative__97f785f4_uniq”
DETAIL: Key (publication_id, relative_path)=(6790107c-dc2f-4375-90d0-0656462a497d, pool/all/f/flashrom/libflashrom-dev_1.2-5_amd64.deb) already exists.

20.04
duplicate key value violates unique constraint “core_publishedartifact_publication_id_relative__97f785f4_uniq”
DETAIL: Key (publication_id, relative_path)=(6790107c-dc2f-4375-90d0-0656462a497d, pool/all/f/flashrom/libflashrom-dev_1.2-5_amd64.deb) already exists.

I already re-made the repo’s but still ended up with the same problem. I also tried running:
yum update
foreman-maintain service stop
foreman-installer
foreman-rake katello:delete_orphaned_content RAILS_ENV=production

But to no avail. The repo’s aren’t 48 hours old yet i believe.

They did all 3 complete but only when i did them indiviually.

I did read about this issue but i’m not sure how to resolve it, i believe there’s some sort of product or two which pulp can’t handle somehow. Everything else works great except this.

Thank you in advance!

Expected outcome:
all 3 products can sync with their upstream withoud any issues (at the same time/when using the same sync plan).

Foreman and Proxy versions:
foreman-3.3.0-1.el8
foreman-proxy-3.3.0-1.el8

Foreman and Proxy plugin versions:

Installed Packages

  • candlepin-4.1.11-1.el8.noarch
  • candlepin-selinux-4.1.11-1.el8.noarch
  • foreman-3.3.0-1.el8.noarch
  • foreman-cli-3.3.0-1.el8.noarch
  • foreman-debug-3.3.0-1.el8.noarch
  • foreman-dynflow-sidekiq-3.3.0-1.el8.noarch
  • foreman-installer-3.3.0-1.el8.noarch
  • foreman-installer-katello-3.3.0-1.el8.noarch
  • foreman-postgresql-3.3.0-1.el8.noarch
  • foreman-proxy-3.3.0-1.el8.noarch
  • foreman-release-3.3.0-1.el8.noarch
  • foreman-selinux-3.3.0-1.el8.noarch
  • foreman-service-3.3.0-1.el8.noarch
  • katello-4.5.0-1.el8.noarch
  • katello-ca-consumer-vcdsat01.virtualcenter.lan-1.0-7.noarch
  • katello-certs-tools-2.9.0-1.el8.noarch
  • katello-client-bootstrap-1.7.9-1.el8.noarch
  • katello-common-4.5.0-1.el8.noarch
  • katello-debug-4.5.0-1.el8.noarch
  • katello-repos-4.5.0-1.el8.noarch
  • katello-selinux-4.0.2-1.el8.noarch
  • pulpcore-selinux-1.3.0-1.el8.x86_64
  • python39-pulp-ansible-0.13.0-3.el8.noarch
  • python39-pulp-certguard-1.5.2-3.el8.noarch
  • python39-pulp-cli-0.14.0-2.el8.noarch
  • python39-pulp-container-2.10.3-4.el8.noarch
  • python39-pulp-deb-2.18.0-3.el8.noarch
  • python39-pulp-file-1.10.2-2.el8.noarch
  • python39-pulp-python-3.7.1-1.el8.noarch
  • python39-pulp-rpm-3.17.7-1.el8.noarch
  • python39-pulpcore-3.18.5-2.el8.noarch
  • qpid-proton-c-0.37.0-1.el8.x86_64
  • rubygem-foreman-tasks-6.0.2-1.fm3_3.el8.noarch
  • rubygem-foreman_ansible-7.1.2-1.fm3_3.el8.noarch
  • rubygem-foreman_azure_rm-2.2.6-3.fm3_3.el8.noarch
  • rubygem-foreman_discovery-21.0.1-2.fm3_3.el8.noarch
  • rubygem-foreman_hooks-0.3.17-3.fm3_3.el8.noarch
  • rubygem-foreman_maintain-1.1.1-3.el8.noarch
  • rubygem-foreman_openscap-5.2.2-2.fm3_3.el8.noarch
  • rubygem-foreman_puppet-4.0.1-1.fm3_3.el8.noarch
  • rubygem-foreman_remote_execution-7.1.1-1.fm3_3.el8.noarch
  • rubygem-foreman_rh_cloud-5.0.35-1.fm3_3.el8.noarch
  • rubygem-hammer_cli-3.3.0-1.el8.noarch
  • rubygem-hammer_cli_foreman-3.3.0-1.el8.noarch
  • rubygem-hammer_cli_foreman_ansible-0.3.4-1.fm3_0.el8.noarch
  • rubygem-hammer_cli_foreman_azure_rm-0.2.2-1.fm3_1.el8.noarch
  • rubygem-hammer_cli_foreman_puppet-0.0.6-1.fm3_3.el8.noarch
  • rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
  • rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
  • rubygem-hammer_cli_foreman_virt_who_configure-0.0.8-1.el8.noarch
  • rubygem-hammer_cli_katello-1.5.2-1.el8.noarch
  • rubygem-katello-4.5.0-1.el8.noarch
  • rubygem-pulp_ansible_client-0.13.1-1.el8.noarch
  • rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
  • rubygem-pulp_container_client-2.10.3-1.el8.noarch
  • rubygem-pulp_deb_client-2.18.0-1.el8.noarch
  • rubygem-pulp_file_client-1.10.0-1.el8.noarch
  • rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
  • rubygem-pulp_python_client-3.6.0-1.el8.noarch
  • rubygem-pulp_rpm_client-3.17.4-1.el8.noarch
  • rubygem-pulpcore_client-3.18.5-1.el8.noarch
  • rubygem-puppetdb_foreman-5.0.0-4.fm3_3.el8.noarch
  • rubygem-qpid_proton-0.37.0-1.el8.x86_64
  • rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch

Distribution and version:
RHEL 8

Other relevant data:
Setup of the products:

This is most likely a bug in pulp_deb. You can open an issue here: Issues · pulp/pulp_deb · GitHub
Be sure to include the pulp_deb version (python39-pulp-deb-2.18.0-3.el8.noarch) as well as a link to this thread.

That being said, I doubt this issue can be prioritized and fixed very quickly, so I recommend trying the following workaround for now:

Try breaking up your Katello repos into several repositories, that sync one APT distribution each. So instead of having one repo with Releases/Distributions: focal forcal-updates focal-backports focal-security, create one Katello repo with Releases/Distributions: focal, and another with Releases/Distributions: focal-updates, and so forth. If breaking up the repos in this way makes the errors go away, then that is also good debugging information.

Heya! First of all thank you for your reply! I changed the products to be the same as ubuntu set’s them, and everything sync’s up perfect :). Had a typo but that’s fixed aswell!

While i don’t think i have the logs for it anymore, it might be possible that if i only keep the 4 unique release/distributions but give it all the components that it might work again. I’ll try that out aswell and from there i’ll submit a bug report :slight_smile: thank you again for ya helpful advice!

Yup confirmed that separating only the release/distro seems to make it all work fine and dandy!

Also i was looking trough the issues but saw this one:

I believe that’s the same bug i’m dealing with?

No they are two different bugs. I think your bug happens when the same package is synced from multiple distributions (within a single sync), and then there is some kind of race condition on creating the one DB entry for the duplicate package. The bug you linked above on the other hand concerns only metadata (not packages), and cannot occur within a single sync.

I am glad you can work around the problem for now, since I don’t think your bug is an easy problem to solve, and is therefore unlikely to be fixed very quickly.

I went and created a pulp_deb issue to track this: https://github.com/pulp/pulp_deb/issues/611

Like I said, I don’t think it can be fixed very quickly, so anyone affected should use the workaround for now.