Delete Orphaned Content fails

Problem:
When trying to delete orphaned content of of the tasks fail and gets stuck in state paused.

$ foreman-rake katello:delete_orphaned_content RAILS_ENV=production
Orphaned content deletion started in background.
Orphaned content deletion started in background.

# sometime later
$ hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused" '
[
  {
    "ID": "54bd571b-a831-4cb2-b14c-bdac9b988e19",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-07-19 16:02:17 UTC",
    "Ended at": null,
    "Duration": "00:04:17.689881",
    "Owner": "foreman_admin",
    "Task errors": [
      "Repository matching query does not exist."
    ]
  }
]

Expected outcome:
Both (all) tasks finish successfully.

Foreman and Proxy versions:

  • foreman - 3.2.1

Foreman and Proxy plugin versions:

  • foreman-tasks - 6.0.1
  • foreman_host_reports - 1.0.2
  • foreman_puppet - 3.0.7
  • foreman_remote_execution - 6.0.0
  • katello - 4.4.1

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:
Pulp href:
(the ids are diferent because the “screenshots” are from different tests, but the error is always the same)

$ http  https://updates.example.com/pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 949
Content-Type: application/json
Correlation-ID: 6faa5061326246618ea42d0455900ac4
Date: Tue, 19 Jul 2022 13:25:38 GMT
Keep-Alive: timeout=15, max=100
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 updates.example.com
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Repository matching query does not exist.",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 380, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py\", line 66, in orphan_cleanup\n    c.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 340, in cleanup_subrepos\n    Variant.objects.filter(repository=subrepo).delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 338, in cleanup_subrepos\n    subrepo = instance.repository\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 187, in __get__\n    rel_obj = self.get_object(instance)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 154, in get_object\n    return qs.get(self.field.get_reverse_related_filter(instance))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 435, in get\n    raise self.model.DoesNotExist(\n"
    },
    "finished_at": "2022-07-19T10:08:40.420909Z",
    "logging_cid": "1f4dc8e6295a4b5881f906d06d56a560",
    "name": "pulpcore.app.tasks.orphan.orphan_cleanup",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "clean-up.content",
            "done": 2000,
            "message": "Clean up orphan Content",
            "state": "running",
            "suffix": null,
            "total": 15379
        }
    ],
    "pulp_created": "2022-07-19T10:08:16.995162Z",
    "pulp_href": "/pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/",
    "reserved_resources_record": [],
    "started_at": "2022-07-19T10:08:21.918340Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/4dda7e72-f009-464a-9235-adf4d4b5f605/"
}

Could you also provide relevant pulp logs from /var/log/messages when you hit this.

Sure, is this enough?

$ grep pulpcore  /var/log/messages | grep 1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d
Jul 19 11:08:17 updates pulpcore-api: pulp [41ede0200fb243a6b731cfdb275655e8]:  - - [19/Jul/2022:10:08:17 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:17 updates pulpcore-api: pulp [cf935327ef3d468ab7f88805e6c67606]:  - - [19/Jul/2022:10:08:17 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:18 updates pulpcore-api: pulp [123e124d40554c7d99d402a0f1afb5a4]:  - - [19/Jul/2022:10:08:18 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:19 updates pulpcore-api: pulp [914343a47d294cb2a0a379c73a06ba02]:  - - [19/Jul/2022:10:08:19 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:19 updates pulpcore-api: pulp [a97d48d489304cbd8cfbdbc4bc88d1a5]:  - - [19/Jul/2022:10:08:19 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:20 updates pulpcore-api: pulp [9cfd9dce3c424225abfc988f3351d392]:  - - [19/Jul/2022:10:08:20 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:21 updates pulpcore-api: pulp [7b340bfcee044fa4828f417267e1ac9c]:  - - [19/Jul/2022:10:08:21 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:21 updates pulpcore-worker-1: pulp [1f4dc8e6295a4b5881f906d06d56a560]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d
Jul 19 11:08:22 updates pulpcore-api: pulp [087fa286e5324cda88db6f4e5538160d]:  - - [19/Jul/2022:10:08:22 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:23 updates pulpcore-api: pulp [67280519d9424015a0a6f6ea03ac753d]:  - - [19/Jul/2022:10:08:23 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:25 updates pulpcore-api: pulp [a9334c17bfc0470bb9cb48f6b2887673]:  - - [19/Jul/2022:10:08:25 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:26 updates pulpcore-api: pulp [9edd2cc3cede4e56b2aa95624fd65020]:  - - [19/Jul/2022:10:08:26 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:28 updates pulpcore-api: pulp [3944fcfd636740ddbeecb970a8aef8c5]:  - - [19/Jul/2022:10:08:28 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:30 updates pulpcore-api: pulp [270555d5b8b74264a2191be9f6ed9dc8]:  - - [19/Jul/2022:10:08:30 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:32 updates pulpcore-api: pulp [460424778acf4aeb93aa0a04fe1ed17e]:  - - [19/Jul/2022:10:08:32 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:35 updates pulpcore-api: pulp [b07d40b6656046749c6ea1264df774bc]:  - - [19/Jul/2022:10:08:35 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:37 updates pulpcore-api: pulp [788458c4f83a44459749b2f4e7908882]:  - - [19/Jul/2022:10:08:37 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:40 updates pulpcore-worker-1: pulp [1f4dc8e6295a4b5881f906d06d56a560]: pulpcore.tasking.pulpcore_worker:INFO: Task 1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d failed (Repository matching query does not exist.)
Jul 19 11:08:41 updates pulpcore-api: pulp [e4f9ce65b8a34ef78bf7385f49dde6bf]:  - - [19/Jul/2022:10:08:41 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 3335 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 14:25:38 updates pulpcore-api: pulp [6faa5061326246618ea42d0455900ac4]:  - - [19/Jul/2022:13:25:38 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 3335 "-" "HTTPie/1.0.3"
Jul 19 14:29:19 updates pulpcore-api: pulp [146f6d4c522c4df283a5ff4748de85e5]:  - - [19/Jul/2022:13:29:19 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 3335 "-" "HTTPie/1.0.3"

Probably not enough, full log here: Jul 19 11:08:17 updates pulpcore-worker-2: pulp [2f70a169d8f6420c82f2731ed23cbe2 - Pastebin.com

Is this the issue?

Jul 19 11:08:30 updates pulpcore-worker-7: pulp [08df98f3d01441c1b876f7bd9277aaa3]: pulpcore.tasking.pulpcore_worker:INFO: Task d4d76c17-cb22-427a-8d87-f5d34250e4dd failed (update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
Jul 19 11:08:30 updates pulpcore-worker-7: DETAIL:  Key (pulp_id)=(3f55a399-e1aa-4db5-a2b5-5b458aaf2552) is still referenced from table "pulp_2to3_migration_pulp2distributor".

We migrated from pulp2 several versions ago.

I think removing Pulp 2 should fix this. Try foreman-maintain content remove-pulp2.

The error is referring to the Pulp 2 - Pulp 3 migration plugin, which is removed during the Pulp 2 removal.

2 Likes

Hello @iballou
When I saw that error i though exactly the same.
Unfortunately after running

$ foreman-maintain content remove-pulp2
$ su - postgres -c 'psql pulpcore -c "\dt pulp_2to3_migration_*"'

Which deleted all the old tables, we still got the same error.

  • messages:
Jul 19 18:17:40 updates pulpcore-api: pulp [29a4aa8b574f485ebe72425ebb6505c5]:  - - [19/Jul/2022:17:17:40 +0000] "POST /pulp/api/v3/orphans/cleanup/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:40 updates pulpcore-worker-3: pulp [52be108736154bedbd8a6c7e9f7bb869]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 898f459f-1f48-40f7-ae07-a650acd3f822
Jul 19 18:17:40 updates pulpcore-worker-3: pulp [52be108736154bedbd8a6c7e9f7bb869]: pulpcore.app.tasks.repository:INFO: Deleting and squashing version 2 of repository 'CentOS_8-CentOS_8_AppStream_RPMS'
Jul 19 18:17:40 updates pulpcore-api: pulp [c9c9b400da7943c0b38db926cea6773e]:  - - [19/Jul/2022:17:17:40 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:40 updates pulpcore-worker-6: pulp [29a4aa8b574f485ebe72425ebb6505c5]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 18ddd0f4-eb2c-4bd1-82b1-5125bd049f95
Jul 19 18:17:40 updates pulpcore-api: pulp [4e03488667314c129be3c18a250e230a]:  - - [19/Jul/2022:17:17:40 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:41 updates pulpcore-api: pulp [2cfd5424ccad4833adb84670fdb9e4ab]:  - - [19/Jul/2022:17:17:41 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:42 updates pulpcore-api: pulp [1620a0dfb4b24d58b4cf85103a57b404]:  - - [19/Jul/2022:17:17:42 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:42 updates pulpcore-api: pulp [1515242e99874bb7980821c7146e501b]:  - - [19/Jul/2022:17:17:42 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:43 updates pulpcore-api: pulp [3c23353536d2450d8870a68f67cbfc01]:  - - [19/Jul/2022:17:17:43 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:43 updates pulpcore-worker-3: pulp [52be108736154bedbd8a6c7e9f7bb869]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 898f459f-1f48-40f7-ae07-a650acd3f822
Jul 19 18:17:44 updates pulpcore-api: pulp [caef3342474844299fb4d656367cf1c4]:  - - [19/Jul/2022:17:17:44 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:45 updates pulpcore-api: pulp [f06c22189fd143169b59c55a912593b2]:  - - [19/Jul/2022:17:17:45 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:46 updates pulpcore-api: pulp [bbb926c590b64d5f98834ba707ef8813]:  - - [19/Jul/2022:17:17:46 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:47 updates pulpcore-api: pulp [1aa4c51b1e504c4c9cf9e723b48875aa]:  - - [19/Jul/2022:17:17:47 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:49 updates pulpcore-api: pulp [5a5d39397dac4db181498e2683e98a6d]:  - - [19/Jul/2022:17:17:49 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:50 updates pulpcore-worker-6: pulp [29a4aa8b574f485ebe72425ebb6505c5]: pulpcore.tasking.pulpcore_worker:INFO: Task 18ddd0f4-eb2c-4bd1-82b1-5125bd049f95 failed (Repository matching query does not exist.)
Jul 19 18:17:50 updates pulpcore-worker-6: pulp [29a4aa8b574f485ebe72425ebb6505c5]: pulpcore.tasking.pulpcore_worker:INFO:   File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in _perform_task
Jul 19 18:17:50 updates pulpcore-worker-6: result = func(*args, **kwargs)
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py", line 66, in orphan_cleanup
Jul 19 18:17:50 updates pulpcore-worker-6: c.delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 746, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: deleted, _rows_count = collector.delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py", line 435, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: signals.post_delete.send(
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 180, in send
Jul 19 18:17:50 updates pulpcore-worker-6: return [
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
Jul 19 18:17:50 updates pulpcore-worker-6: (receiver, receiver(signal=self, sender=sender, **named))
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py", line 340, in cleanup_subrepos
Jul 19 18:17:50 updates pulpcore-worker-6: Variant.objects.filter(repository=subrepo).delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 746, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: deleted, _rows_count = collector.delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py", line 435, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: signals.post_delete.send(
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 180, in send
Jul 19 18:17:50 updates pulpcore-worker-6: return [
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
Jul 19 18:17:50 updates pulpcore-worker-6: (receiver, receiver(signal=self, sender=sender, **named))
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py", line 338, in cleanup_subrepos
Jul 19 18:17:50 updates pulpcore-worker-6: subrepo = instance.repository
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
Jul 19 18:17:50 updates pulpcore-worker-6: rel_obj = self.get_object(instance)
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
Jul 19 18:17:50 updates pulpcore-worker-6: return qs.get(self.field.get_reverse_related_filter(instance))
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
Jul 19 18:17:50 updates pulpcore-worker-6: raise self.model.DoesNotExist(
Jul 19 18:17:51 updates pulpcore-api: pulp [4fd22c711ce248d1a5348a0e393a4f8f]:  - - [19/Jul/2022:17:17:51 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 3335 "-" "OpenAPI-Generator/3.16.7/ruby"
  • paused
$ hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused" '
[
  {
    "ID": "9057c237-d82c-410b-af81-701f917b7e16",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-07-19 17:16:58 UTC",
    "Ended at": null,
    "Duration": "00:01:41.365808",
    "Owner": "foreman_admin",
    "Task errors": [
      "Repository matching query does not exist."
    ]
  }
]
  • pulp href
# http_pulp /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 943
Content-Type: application/json
Correlation-ID: 4ca178df822b496a96a30bc5c3864909
Date: Tue, 19 Jul 2022 17:20:25 GMT
Keep-Alive: timeout=15, max=100
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 updates.example.com
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Repository matching query does not exist.",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 380, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py\", line 66, in orphan_cleanup\n    c.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 340, in cleanup_subrepos\n    Variant.objects.filter(repository=subrepo).delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 338, in cleanup_subrepos\n    subrepo = instance.repository\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 187, in __get__\n    rel_obj = self.get_object(instance)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 154, in get_object\n    return qs.get(self.field.get_reverse_related_filter(instance))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 435, in get\n    raise self.model.DoesNotExist(\n"
    },
    "finished_at": "2022-07-19T17:17:50.928953Z",
    "logging_cid": "29a4aa8b574f485ebe72425ebb6505c5",
    "name": "pulpcore.app.tasks.orphan.orphan_cleanup",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "clean-up.content",
            "done": 2000,
            "message": "Clean up orphan Content",
            "state": "running",
            "suffix": null,
            "total": 15379
        }
    ],
    "pulp_created": "2022-07-19T17:17:40.032014Z",
    "pulp_href": "/pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/",
    "reserved_resources_record": [],
    "started_at": "2022-07-19T17:17:40.292569Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/38481210-0d06-4c37-96b2-20d50e237b6a/"
}

I have a feeling this may be a new Pulp issue. Would you mind filing an issue here (Issues · pulp/pulpcore · GitHub) with the backtrace and the task information?

Because we weren’t on the latest version we upgraded to foreman-3.3.0 and katello-4.5.0 but we still got the same error.
So we opened an issue upstream: https://github.com/pulp/pulpcore/issues/2996

1 Like

I was able to reproduce and fix this, patch incoming.