Sync fails for Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9

Problem:
Today, the sync of the repository Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9 from RedHat failed with an exception

Katello::Errors::Pulp3Error: duplicate key value violates unique constraint "core_publishedartifact_publication_id_relative__97f785f4_uniq"
DETAIL:  Key (publication_id, relative_path)=(019cdb92-ab7f-74f5-9bd4-dbb74442d8fa, Packages/g/galera-26.4.20-1.module+el9.5.0+22578+dc54e53f.x86_64.rpm) already exists.

I have also tried a complete sync, with the same result.

Expected outcome:
Successful sync.

Foreman and Proxy versions:
katello-4.18.1-1.el9.noarch
foreman-3.16.2-1.el9.noarch

Distribution and version:
AlmaLinux 9.7

Other relevant data:

Action:

Actions::Pulp3::Repository::Sync

Input:

{"repo_id"=>101581,
 "smart_proxy_id"=>1,
 "options"=>{"optimize"=>false},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"51e00235-a0b8-43a7-b4aa-1389fd724ae8",
 "current_timezone"=>"Europe/Berlin",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>5}

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/019cdb8d-f797-7fc1-bdcf-25b73b9928f1/",
    "prn"=>"prn:core.task:019cdb8d-f797-7fc1-bdcf-25b73b9928f1",
    "pulp_created"=>"2026-03-11T06:20:40.215Z",
    "pulp_last_updated"=>"2026-03-11T06:20:40.215Z",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"51e00235-a0b8-43a7-b4aa-1389fd724ae8",
    "created_by"=>"/pulp/api/v3/users/1/",
    "unblocked_at"=>"2026-03-11T06:20:40.271Z",
    "started_at"=>"2026-03-11T06:20:40.325Z",
    "finished_at"=>"2026-03-11T06:25:05.670Z",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.12/site-packages/pulpcore/tasking/tasks.py\", line 71, in _execute_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "             ^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.12/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 603, in synchronize\n" +
       "    add_metadata_to_publication(publication, repo_version, prefix=path)\n" +
       "  File \"/usr/lib/python3.12/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 211, in add_metadata_to_publication\n" +
       "    PublishedArtifact.objects.bulk_create(published_artifacts)\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/models/manager.py\", line 87, in manager_method\n" +
       "    return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
       "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/models/query.py\", line 786, in bulk_create\n" +
       "    returned_columns = self._batched_insert(\n" +
       "                       ^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/models/query.py\", line 1831, in _batched_insert\n" +
       "    self._insert(\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/models/query.py\", line 1805, in _insert\n" +
       "    return query.get_compiler(using=using).execute_sql(returning_fields)\n" +
       "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/models/sql/compiler.py\", line 1822, in execute_sql\n" +
       "    cursor.execute(sql, params)\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/backends/utils.py\", line 67, in execute\n" +
       "    return self._execute_with_wrappers(\n" +
       "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/backends/utils.py\", line 80, in _execute_with_wrappers\n" +
       "    return executor(sql, params, many, context)\n" +
       "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/backends/utils.py\", line 84, in _execute\n" +
       "    with self.db.wrap_database_errors:\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/utils.py\", line 91, in __exit__\n" +
       "    raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
       "  File \"/usr/lib/python3.12/site-packages/django/db/backends/utils.py\", line 89, in _execute\n" +
       "    return self.cursor.execute(sql, params)\n" +
       "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n" +
       "  File \"/usr/lib/python3.12/site-packages/psycopg/cursor.py\", line 97, in execute\n" +
       "    raise ex.with_traceback(None)\n",
      "description"=>
       "duplicate key value violates unique constraint \"core_publishedartifact_publication_id_relative__97f785f4_uniq\"\n" +
       "DETAIL:  Key (publication_id, relative_path)=(019cdb92-ab7f-74f5-9bd4-dbb74442d8fa, Packages/g/galera-26.4.20-1.module+el9.5.0+22578+dc54e53f.x86_64.rpm) already exists."},
    "worker"=>"/pulp/api/v3/workers/019cc1d3-02d1-753a-95df-8e45b6d96df7/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>11,
       "suffix"=>nil},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"completed",
       "done"=>0,
       "suffix"=>nil},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"completed",
       "done"=>0,
       "suffix"=>nil},
      {"message"=>"Parsed Modulemd",
       "code"=>"sync.parsing.modulemds",
       "state"=>"completed",
       "total"=>123,
       "done"=>123,
       "suffix"=>nil},
      {"message"=>"Parsed Modulemd-defaults",
       "code"=>"sync.parsing.modulemd_defaults",
       "state"=>"completed",
       "total"=>9,
       "done"=>9,
       "suffix"=>nil},
      {"message"=>"Parsed Modulemd Obsolete",
       "code"=>"sync.parsing.modulemd_obsoletes",
       "state"=>"completed",
       "total"=>0,
       "done"=>0,
       "suffix"=>nil},
      {"message"=>"Skipping Packages",
       "code"=>"sync.skipped.packages",
       "state"=>"completed",
       "total"=>1,
       "done"=>1,
       "suffix"=>nil},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"completed",
       "total"=>30980,
       "done"=>30980,
       "suffix"=>nil},
      {"message"=>"Parsed Comps",
       "code"=>"sync.parsing.comps",
       "state"=>"completed",
       "total"=>72,
       "done"=>72,
       "suffix"=>nil},
      {"message"=>"Parsed Advisories",
       "code"=>"sync.parsing.advisories",
       "state"=>"completed",
       "total"=>5850,
       "done"=>5850,
       "suffix"=>nil},
      {"message"=>"Un-Associating Content",
       "code"=>"unassociating.content",
       "state"=>"completed",
       "done"=>0,
       "suffix"=>nil}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["prn:rpm.rpmrepository:df928266-53d6-4f39-a836-5a6051deb567",
      "shared:prn:rpm.rpmremote:ab8724e2-a8d1-4efd-8254-ba0b69860286",
      "shared:prn:core.domain:018d10eb-22f7-72a8-ab19-6f036c81f631"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>37, "failed"=>1}}

Exception:

Katello::Errors::Pulp3Error: duplicate key value violates unique constraint "core_publishedartifact_publication_id_relative__97f785f4_uniq"
DETAIL:  Key (publication_id, relative_path)=(019cdb92-ab7f-74f5-9bd4-dbb74442d8fa, Packages/g/galera-26.4.20-1.module+el9.5.0+22578+dc54e53f.x86_64.rpm) already exists.

Backtrace:

/usr/share/gems/gems/katello-4.18.1/app/lib/actions/pulp3/abstract_async_task.rb:107:in `block in check_for_errors'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/pulp3/abstract_async_task.rb:105:in `each'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/pulp3/abstract_async_task.rb:105:in `check_for_errors'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/pulp3/abstract_async_task.rb:161:in `poll_external_task'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action/polling.rb:22:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action/cancellable.rb:15:in `run'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:33:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.18.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.18.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in `wrap'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action/progress.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:33:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-11.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:588:in `catch'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:588:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:491:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:491:in `catch'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:491:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:583:in `execute_run'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/action.rb:304:in `execute'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/director.rb:95:in `execute'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
/usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'
/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
1 Like

Hi @gvde ,

You’re hitting a RHEL metadata issue that’s been causing Pulp sync errors across many many environments: https://issues.redhat.com/browse/SAT-43249

It’s being investigated on the CDN side from what I’ve heard. The link I shared up above is a Satellite Jira - I’m not sure if there is a Pulp fix to avoid this issue in the future, but we’ll see.

4 Likes