Unable to publish new version of content view with filter

Support
1

Hi,

I have issue with publishing new version of content view.
I get such error:

duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"
DETAIL:  Key (repository_id, content_id, version_removed_id)=(3670186b-a4d7-43df-8f08-443efd45e07f, cdfba66d-0fcb-4a60-8f6c-976d7ed38c0e, 81f3baba-c5f0-4a93-a45b-8878d321acc7) already exists.

There are filters for yum repositories, but when I remove these filteres everything is ok.
When I copy this CV with new name everything is ok - I can publish new version with old filters.

I think this error could be related with incomplete removal of older version of content view.

Is there any tool to remove garbage of this incomplete removal or any advice how to deal with this issue?

Katello 4.0.1
Foreman 2.4

2

I have exactly the same issue when publishing RHEL8 CV.
Katello 4.1.3
Foreman 2.5.3

3

edit: added a single rpm package to the filter. Publishing Content View did work. Also on multiple attempts.

4

Would you happen to have the stack trace by any chance from /var/log/foreman/production.log ?

5

Adding onto @Partha_Aji 's comment, can we see the dynflow task information around the error?

Also, do you have any paused tasks that errored out relating to content views or maybe repositories?

6

The task paused, then I skipped the “MultiCopyContent” subtask via Dynflow console. So the task finished with success. The repository sync is fine. I haven’t had any issues there.
My RHEL 8 CV contains 4 repositories:

  • Red Hat CodeReady Linux Builder for RHEL 8 x86_64 RPMs 8
  • Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8
  • Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs 8
  • Red Hat Satellite Tools 6.6 for RHEL 8 x86_64 RPMs

Dynflow task output:

Id: 0b89630e-f6ad-466b-b6a9-638999c47897

Label: Actions::Katello::ContentView::Publish

Status: stopped

Result: warning

Started at: 2021-09-09 10:03:08 UTC

Ended at: 2021-09-09 10:20:32 UTC

Plan
Run
Finalize
Execution History
7: Actions::Pulp3::Repository::Create (success) [ 0.20s / 0.20s ]
9: Actions::Pulp3::Repository::SaveVersion (success) [ 0.28s / 0.28s ]
10: Actions::Katello::Repository::Create (success) [ 0.06s / 0.06s ]
14: Actions::Pulp3::Repository::Create (success) [ 0.20s / 0.20s ]
16: Actions::Pulp3::Repository::SaveVersion (success) [ 0.32s / 0.32s ]
17: Actions::Katello::Repository::Create (success) [ 0.03s / 0.03s ]
21: Actions::Pulp3::Repository::Create (success) [ 0.16s / 0.16s ]
23: Actions::Pulp3::Repository::SaveVersion (success) [ 0.29s / 0.29s ]
24: Actions::Katello::Repository::Create (success) [ 0.06s / 0.06s ]
28: Actions::Pulp3::Repository::Create (success) [ 0.07s / 0.07s ]
30: Actions::Pulp3::Repository::SaveVersion (success) [ 0.27s / 0.27s ]
31: Actions::Katello::Repository::Create (success) [ 0.06s / 0.06s ]
36: Actions::Pulp3::Repository::MultiCopyContent (skipped) [ 928.22s / 36.63s ]
Queue: default

Started at: 2021-09-09 10:03:12 UTC

Ended at: 2021-09-09 10:18:40 UTC

Real time: 928.22s

Execution time (excluding suspended state): 36.63s

Input:

---
solve_dependencies: false
repo_id_map:
  "[5]":
    dest_repo: 23757
    filter_ids: []
  "[6]":
    dest_repo: 23758
    filter_ids: []
  "[9]":
    dest_repo: 23759
    filter_ids: []
  "[5856]":
    dest_repo: 23760
    filter_ids:
    - 16
smart_proxy_id: 1
remote_user: admin
remote_cp_user: admin
current_request_id: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
current_timezone: Europe/Berlin
current_organization_id: 1
current_location_id: 
current_user_id: 27
Output:

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/db1c311a-a744-40b6-a204-66ab51cdfdec/"
  pulp_created: '2021-09-09T10:03:42.126+00:00'
  state: failed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.168+00:00'
  finished_at: '2021-09-09T10:03:43.398+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py", line 218, in add_and_remove
          new_version.remove_content(models.Content.objects.filter(pk__in=remove_content_units))
        File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 796, in remove_content
          q_set.update(version_removed=self)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 741, in update
          rows = query.get_compiler(self.db).execute_sql(CURSOR)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql
          cursor = super().execute_sql(result_type)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"
      DETAIL:  Key (repository_id, content_id, version_removed_id)=(1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d, 995a0ffa-ad9c-45ab-b44e-6fb972ab5405, 8d4d8dc8-df24-490e-9d62-168e4cbcd66d) already exists.
  worker: "/pulp/api/v3/workers/7a7376d4-ae32-4cd0-8774-407fb4bfc4b1/"
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
- pulp_href: "/pulp/api/v3/tasks/ce298bd4-9914-4e45-994e-d9ac75d4cd2b/"
  pulp_created: '2021-09-09T10:03:42.191+00:00'
  state: running
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:43.440+00:00'
  worker: "/pulp/api/v3/workers/ec059f4c-140e-4075-9ef6-d406fcedcb4b/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - ''
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
- pulp_href: "/pulp/api/v3/tasks/c182237f-5c93-4e3d-9b2a-9062ee7f79cb/"
  pulp_created: '2021-09-09T10:03:42.291+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.336+00:00'
  finished_at: '2021-09-09T10:03:44.678+00:00'
  worker: "/pulp/api/v3/workers/172f43a1-996b-4aea-9a76-ac992e3894d8/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/versions/29/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
- pulp_href: "/pulp/api/v3/tasks/a4579e8e-71ac-4771-ac36-501ed6e6f847/"
  pulp_created: '2021-09-09T10:03:42.362+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:44.736+00:00'
  finished_at: '2021-09-09T10:03:45.129+00:00'
  worker: "/pulp/api/v3/workers/172f43a1-996b-4aea-9a76-ac992e3894d8/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/versions/30/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
- pulp_href: "/pulp/api/v3/tasks/9d5fef4c-2bcd-4d48-be17-e7956354208f/"
  pulp_created: '2021-09-09T10:03:42.471+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.529+00:00'
  finished_at: '2021-09-09T10:03:42.729+00:00'
  worker: "/pulp/api/v3/workers/ec059f4c-140e-4075-9ef6-d406fcedcb4b/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/versions/6/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
- pulp_href: "/pulp/api/v3/tasks/bbce3145-0b85-4dfc-9d61-67c40f05368b/"
  pulp_created: '2021-09-09T10:03:42.581+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.671+00:00'
  finished_at: '2021-09-09T10:03:45.371+00:00'
  worker: "/pulp/api/v3/workers/2a3cfeb4-faf4-4504-9c04-f296a99ede6c/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/versions/14/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
- pulp_href: "/pulp/api/v3/tasks/159f4aa2-9525-46b0-9dc4-5d334f589237/"
  pulp_created: '2021-09-09T10:03:44.566+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/ec69f182-aa0f-4b80-bf78-0e8fc4d0cf35/"
  pulp_created: '2021-09-09T10:03:46.451+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/3f3d49cc-fc15-41d2-ac12-ab59e3aa8f6d/"
  pulp_created: '2021-09-09T10:03:46.871+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/c314cb1e-4fc1-4bab-b6f5-8a75f03f4442/"
  pulp_created: '2021-09-09T10:03:48.513+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/fc2fb04a-ac9b-4087-833c-d4e5f2f51db4/"
  pulp_created: '2021-09-09T10:03:48.633+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/cd15cff0-b704-4909-bb59-5960a7658c81/"
  pulp_created: '2021-09-09T10:03:48.883+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
task_groups: []
Error:

Katello::Errors::Pulp3Error

duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq" DETAIL: Key (repository_id, content_id, version_removed_id)=(1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d, 995a0ffa-ad9c-45ab-b44e-6fb972ab5405, 8d4d8dc8-df24-490e-9d62-168e4cbcd66d) already exists.

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/pulp3/abstract_async_task.rb:102:in
  `block in check_for_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/pulp3/abstract_async_task.rb:100:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/pulp3/abstract_async_task.rb:100:in
  `check_for_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/pulp3/abstract_async_task.rb:113:in
  `external_task='"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:84:in
  `initiate_external_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:19:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/cancellable.rb:14:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/pulp3/abstract_async_task.rb:10:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.3/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in
  `wrap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:31:in
  `with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:17:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.4/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/world.rb:31:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:571:in
  `block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in
  `block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in
  `block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in
  `with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:565:in
  `execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:286:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director.rb:69:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors.rb:18:in
  `run_user_code'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in
  `execute_job'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in
  `block (2 levels) in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in
  `block in invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in
  `invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in
  `block in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in
  `block (6 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in
  `local'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in
  `block (5 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in
  `block in <module:Sidekiq>'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in
  `block (4 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in
  `stats'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in
  `block (3 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in
  `block (2 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in
  `global'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in
  `block in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in
  `with_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in
  `with_job_hash_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in
  `dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in
  `process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in
  `process_one'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in
  `watchdog'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in
  `block in safe_thread'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in
  `block in create_with_logging_context'"
38: Actions::Pulp3::Repository::SaveVersions (skipped) [ 0.00s / 0.00s ]
41: Actions::Pulp3::Orchestration::Repository::GenerateMetadata (success) [ 0.26s / 0.26s ]
43: Actions::Pulp3::ContentGuard::Refresh (success) [ 39.04s / 39.04s ]
46: Actions::Pulp3::Orchestration::Repository::GenerateMetadata (success) [ 0.26s / 0.26s ]
48: Actions::Pulp3::ContentGuard::Refresh (success) [ 20.66s / 20.66s ]
51: Actions::Pulp3::Orchestration::Repository::GenerateMetadata (success) [ 0.26s / 0.26s ]
53: Actions::Pulp3::ContentGuard::Refresh (success) [ 7.95s / 7.95s ]
57: Actions::Pulp3::Repository::CreatePublication (success) [ 78.70s / 10.66s ]
59: Actions::Pulp3::Repository::SavePublication (success) [ 0.06s / 0.06s ]
61: Actions::Pulp3::ContentGuard::Refresh (success) [ 5.77s / 5.77s ]
63: Actions::Katello::Repository::IndexContent (success) [ 107.05s / 107.05s ]
65: Actions::Katello::Repository::IndexContent (success) [ 86.03s / 86.03s ]
67: Actions::Katello::Repository::IndexContent (success) [ 72.17s / 72.17s ]
69: Actions::Katello::Repository::IndexContent (success) [ 78.19s / 78.19s ]
73: Actions::Katello::Repository::CheckMatchingContent (success) [ 0.36s / 0.36s ]
76: Actions::Pulp3::Orchestration::Repository::GenerateMetadata (success) [ 0.06s / 0.06s ]
78: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.19s / 0.19s ]
81: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.21s / 0.21s ]
82: Actions::Pulp3::Repository::RefreshDistribution (success) [ 1.80s / 0.53s ]
84: Actions::Pulp3::Repository::SaveDistributionReferences (success) [ 0.11s / 0.11s ]
87: Actions::Katello::Repository::IndexContent (success) [ 1.60s / 1.60s ]
91: Actions::Katello::Repository::CheckMatchingContent (success) [ 0.21s / 0.21s ]
94: Actions::Pulp3::Orchestration::Repository::GenerateMetadata (success) [ 0.10s / 0.10s ]
96: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.18s / 0.18s ]
99: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.17s / 0.17s ]
100: Actions::Pulp3::Repository::RefreshDistribution (success) [ 1.08s / 0.50s ]
102: Actions::Pulp3::Repository::SaveDistributionReferences (success) [ 0.18s / 0.18s ]
105: Actions::Katello::Repository::IndexContent (success) [ 0.54s / 0.54s ]
109: Actions::Katello::Repository::CheckMatchingContent (success) [ 0.14s / 0.14s ]
112: Actions::Pulp3::Orchestration::Repository::GenerateMetadata (success) [ 0.05s / 0.05s ]
114: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.18s / 0.18s ]
117: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.24s / 0.24s ]
118: Actions::Pulp3::Repository::RefreshDistribution (success) [ 1.91s / 0.42s ]
120: Actions::Pulp3::Repository::SaveDistributionReferences (success) [ 0.11s / 0.11s ]
123: Actions::Katello::Repository::IndexContent (success) [ 0.23s / 0.23s ]
127: Actions::Katello::Repository::CheckMatchingContent (success) [ 0.19s / 0.19s ]
130: Actions::Pulp3::Orchestration::Repository::GenerateMetadata (success) [ 0.15s / 0.15s ]
132: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.21s / 0.21s ]
135: Actions::Pulp3::ContentGuard::Refresh (success) [ 0.16s / 0.16s ]
136: Actions::Pulp3::Repository::RefreshDistribution (success) [ 1.12s / 0.43s ]
138: Actions::Pulp3::Repository::SaveDistributionReferences (success) [ 0.21s / 0.21s ]
141: Actions::Katello::Repository::IndexContent (success) [ 0.38s / 0.38s ]
147: Actions::Katello::ContentView::ErrataMail (success) [ 0.11s / 0.11s ]
149: Actions::Katello::ContentView::Publish (success) [ 0.03s / 0.03s ]
7

@trendyandy, that might be a pulpcore bug. Can I see your Pulp versions?

sudo pip3 list | grep pulp
8

@iballou , my pulp versions:

root@foreman01 /root # pip3 list --format=columns| grep pulp
pulp-2to3-migration             0.12.0
pulp-ansible                    0.9.0
pulp-certguard                  1.4.0
pulp-container                  2.7.1
pulp-deb                        2.14.1
pulp-file                       1.8.2
pulp-rpm                        3.14.2
pulpcore                        3.14.5
9

Thanks, we’ll get a bug filed and see if we can find a workaround.

10

Here’s the bug: Issue #9396: Duplicate key error during pulpcore.app.tasks.repository.add_and_remove - Pulp

11

Is there a workaround for this issue? I, too, ran into it after a RHEL repo sync process hung and I had to force cancel it. Now trying to delete the content view version that included this repo throws this duplicate key error. I can’t disable/delete the repo either because it’s being used in a content view (which I can’t delete either).

12

@radekb @cbcbcb I have a question - we think we might have pinned this bug down to something in the migration process from Pulp 2, but we are not sure. Are your systems migrated from a Pulp 2 installation or is occurring on a fresh Pulp 3 instance?

13

I just recently starting using this Foreman server and don’t know the history. The main admin is out this week. I can get an answer for you when he returns next week.

14

Just spoke with the Foreman admin and he doesn’t remember the specific Pulp version history, etc. Sorry I can’t be more helpful.

15

No problem, we did figure out the cause and we’re working on the fix.