Delete Orphaned Content fails

mindo · July 19, 2022, 4:23pm

Problem:
When trying to delete orphaned content of of the tasks fail and gets stuck in state paused.

$ foreman-rake katello:delete_orphaned_content RAILS_ENV=production
Orphaned content deletion started in background.
Orphaned content deletion started in background.

# sometime later
$ hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused" '
[
  {
    "ID": "54bd571b-a831-4cb2-b14c-bdac9b988e19",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-07-19 16:02:17 UTC",
    "Ended at": null,
    "Duration": "00:04:17.689881",
    "Owner": "foreman_admin",
    "Task errors": [
      "Repository matching query does not exist."
    ]
  }
]

Expected outcome:
Both (all) tasks finish successfully.

Foreman and Proxy versions:

foreman - 3.2.1

Foreman and Proxy plugin versions:

foreman-tasks - 6.0.1
foreman_host_reports - 1.0.2
foreman_puppet - 3.0.7
foreman_remote_execution - 6.0.0
katello - 4.4.1

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:
Pulp href:
(the ids are diferent because the “screenshots” are from different tests, but the error is always the same)

$ http  https://updates.example.com/pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 949
Content-Type: application/json
Correlation-ID: 6faa5061326246618ea42d0455900ac4
Date: Tue, 19 Jul 2022 13:25:38 GMT
Keep-Alive: timeout=15, max=100
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 updates.example.com
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Repository matching query does not exist.",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 380, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py\", line 66, in orphan_cleanup\n    c.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 340, in cleanup_subrepos\n    Variant.objects.filter(repository=subrepo).delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 338, in cleanup_subrepos\n    subrepo = instance.repository\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 187, in __get__\n    rel_obj = self.get_object(instance)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 154, in get_object\n    return qs.get(self.field.get_reverse_related_filter(instance))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 435, in get\n    raise self.model.DoesNotExist(\n"
    },
    "finished_at": "2022-07-19T10:08:40.420909Z",
    "logging_cid": "1f4dc8e6295a4b5881f906d06d56a560",
    "name": "pulpcore.app.tasks.orphan.orphan_cleanup",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "clean-up.content",
            "done": 2000,
            "message": "Clean up orphan Content",
            "state": "running",
            "suffix": null,
            "total": 15379
        }
    ],
    "pulp_created": "2022-07-19T10:08:16.995162Z",
    "pulp_href": "/pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/",
    "reserved_resources_record": [],
    "started_at": "2022-07-19T10:08:21.918340Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/4dda7e72-f009-464a-9235-adf4d4b5f605/"
}

sajha · July 19, 2022, 4:41pm

Could you also provide relevant pulp logs from /var/log/messages when you hit this.

mindo · July 19, 2022, 4:45pm

Sure, is this enough?

$ grep pulpcore  /var/log/messages | grep 1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d
Jul 19 11:08:17 updates pulpcore-api: pulp [41ede0200fb243a6b731cfdb275655e8]:  - - [19/Jul/2022:10:08:17 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:17 updates pulpcore-api: pulp [cf935327ef3d468ab7f88805e6c67606]:  - - [19/Jul/2022:10:08:17 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:18 updates pulpcore-api: pulp [123e124d40554c7d99d402a0f1afb5a4]:  - - [19/Jul/2022:10:08:18 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:19 updates pulpcore-api: pulp [914343a47d294cb2a0a379c73a06ba02]:  - - [19/Jul/2022:10:08:19 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:19 updates pulpcore-api: pulp [a97d48d489304cbd8cfbdbc4bc88d1a5]:  - - [19/Jul/2022:10:08:19 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:20 updates pulpcore-api: pulp [9cfd9dce3c424225abfc988f3351d392]:  - - [19/Jul/2022:10:08:20 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:21 updates pulpcore-api: pulp [7b340bfcee044fa4828f417267e1ac9c]:  - - [19/Jul/2022:10:08:21 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:21 updates pulpcore-worker-1: pulp [1f4dc8e6295a4b5881f906d06d56a560]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d
Jul 19 11:08:22 updates pulpcore-api: pulp [087fa286e5324cda88db6f4e5538160d]:  - - [19/Jul/2022:10:08:22 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:23 updates pulpcore-api: pulp [67280519d9424015a0a6f6ea03ac753d]:  - - [19/Jul/2022:10:08:23 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:25 updates pulpcore-api: pulp [a9334c17bfc0470bb9cb48f6b2887673]:  - - [19/Jul/2022:10:08:25 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:26 updates pulpcore-api: pulp [9edd2cc3cede4e56b2aa95624fd65020]:  - - [19/Jul/2022:10:08:26 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:28 updates pulpcore-api: pulp [3944fcfd636740ddbeecb970a8aef8c5]:  - - [19/Jul/2022:10:08:28 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:30 updates pulpcore-api: pulp [270555d5b8b74264a2191be9f6ed9dc8]:  - - [19/Jul/2022:10:08:30 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:32 updates pulpcore-api: pulp [460424778acf4aeb93aa0a04fe1ed17e]:  - - [19/Jul/2022:10:08:32 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:35 updates pulpcore-api: pulp [b07d40b6656046749c6ea1264df774bc]:  - - [19/Jul/2022:10:08:35 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:37 updates pulpcore-api: pulp [788458c4f83a44459749b2f4e7908882]:  - - [19/Jul/2022:10:08:37 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 11:08:40 updates pulpcore-worker-1: pulp [1f4dc8e6295a4b5881f906d06d56a560]: pulpcore.tasking.pulpcore_worker:INFO: Task 1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d failed (Repository matching query does not exist.)
Jul 19 11:08:41 updates pulpcore-api: pulp [e4f9ce65b8a34ef78bf7385f49dde6bf]:  - - [19/Jul/2022:10:08:41 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 3335 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 14:25:38 updates pulpcore-api: pulp [6faa5061326246618ea42d0455900ac4]:  - - [19/Jul/2022:13:25:38 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 3335 "-" "HTTPie/1.0.3"
Jul 19 14:29:19 updates pulpcore-api: pulp [146f6d4c522c4df283a5ff4748de85e5]:  - - [19/Jul/2022:13:29:19 +0000] "GET /pulp/api/v3/tasks/1d3a172b-c9a4-4f7a-af8e-d0d7f99c639d/ HTTP/1.1" 200 3335 "-" "HTTPie/1.0.3"

mindo · July 19, 2022, 4:53pm

Probably not enough, full log here: Jul 19 11:08:17 updates pulpcore-worker-2: pulp [2f70a169d8f6420c82f2731ed23cbe2 - Pastebin.com

Is this the issue?

Jul 19 11:08:30 updates pulpcore-worker-7: pulp [08df98f3d01441c1b876f7bd9277aaa3]: pulpcore.tasking.pulpcore_worker:INFO: Task d4d76c17-cb22-427a-8d87-f5d34250e4dd failed (update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
Jul 19 11:08:30 updates pulpcore-worker-7: DETAIL:  Key (pulp_id)=(3f55a399-e1aa-4db5-a2b5-5b458aaf2552) is still referenced from table "pulp_2to3_migration_pulp2distributor".

We migrated from pulp2 several versions ago.

iballou · July 19, 2022, 5:03pm

I think removing Pulp 2 should fix this. Try foreman-maintain content remove-pulp2.

The error is referring to the Pulp 2 - Pulp 3 migration plugin, which is removed during the Pulp 2 removal.

mindo · July 19, 2022, 5:25pm

Hello @iballou
When I saw that error i though exactly the same.
Unfortunately after running

$ foreman-maintain content remove-pulp2
$ su - postgres -c 'psql pulpcore -c "\dt pulp_2to3_migration_*"'

Which deleted all the old tables, we still got the same error.

messages:

Jul 19 18:17:40 updates pulpcore-api: pulp [29a4aa8b574f485ebe72425ebb6505c5]:  - - [19/Jul/2022:17:17:40 +0000] "POST /pulp/api/v3/orphans/cleanup/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:40 updates pulpcore-worker-3: pulp [52be108736154bedbd8a6c7e9f7bb869]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 898f459f-1f48-40f7-ae07-a650acd3f822
Jul 19 18:17:40 updates pulpcore-worker-3: pulp [52be108736154bedbd8a6c7e9f7bb869]: pulpcore.app.tasks.repository:INFO: Deleting and squashing version 2 of repository 'CentOS_8-CentOS_8_AppStream_RPMS'
Jul 19 18:17:40 updates pulpcore-api: pulp [c9c9b400da7943c0b38db926cea6773e]:  - - [19/Jul/2022:17:17:40 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 428 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:40 updates pulpcore-worker-6: pulp [29a4aa8b574f485ebe72425ebb6505c5]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 18ddd0f4-eb2c-4bd1-82b1-5125bd049f95
Jul 19 18:17:40 updates pulpcore-api: pulp [4e03488667314c129be3c18a250e230a]:  - - [19/Jul/2022:17:17:40 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:41 updates pulpcore-api: pulp [2cfd5424ccad4833adb84670fdb9e4ab]:  - - [19/Jul/2022:17:17:41 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:42 updates pulpcore-api: pulp [1620a0dfb4b24d58b4cf85103a57b404]:  - - [19/Jul/2022:17:17:42 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:42 updates pulpcore-api: pulp [1515242e99874bb7980821c7146e501b]:  - - [19/Jul/2022:17:17:42 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 627 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:43 updates pulpcore-api: pulp [3c23353536d2450d8870a68f67cbfc01]:  - - [19/Jul/2022:17:17:43 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:43 updates pulpcore-worker-3: pulp [52be108736154bedbd8a6c7e9f7bb869]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 898f459f-1f48-40f7-ae07-a650acd3f822
Jul 19 18:17:44 updates pulpcore-api: pulp [caef3342474844299fb4d656367cf1c4]:  - - [19/Jul/2022:17:17:44 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:45 updates pulpcore-api: pulp [f06c22189fd143169b59c55a912593b2]:  - - [19/Jul/2022:17:17:45 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:46 updates pulpcore-api: pulp [bbb926c590b64d5f98834ba707ef8813]:  - - [19/Jul/2022:17:17:46 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:47 updates pulpcore-api: pulp [1aa4c51b1e504c4c9cf9e723b48875aa]:  - - [19/Jul/2022:17:17:47 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:49 updates pulpcore-api: pulp [5a5d39397dac4db181498e2683e98a6d]:  - - [19/Jul/2022:17:17:49 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 19 18:17:50 updates pulpcore-worker-6: pulp [29a4aa8b574f485ebe72425ebb6505c5]: pulpcore.tasking.pulpcore_worker:INFO: Task 18ddd0f4-eb2c-4bd1-82b1-5125bd049f95 failed (Repository matching query does not exist.)
Jul 19 18:17:50 updates pulpcore-worker-6: pulp [29a4aa8b574f485ebe72425ebb6505c5]: pulpcore.tasking.pulpcore_worker:INFO:   File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in _perform_task
Jul 19 18:17:50 updates pulpcore-worker-6: result = func(*args, **kwargs)
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py", line 66, in orphan_cleanup
Jul 19 18:17:50 updates pulpcore-worker-6: c.delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 746, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: deleted, _rows_count = collector.delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py", line 435, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: signals.post_delete.send(
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 180, in send
Jul 19 18:17:50 updates pulpcore-worker-6: return [
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
Jul 19 18:17:50 updates pulpcore-worker-6: (receiver, receiver(signal=self, sender=sender, **named))
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py", line 340, in cleanup_subrepos
Jul 19 18:17:50 updates pulpcore-worker-6: Variant.objects.filter(repository=subrepo).delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 746, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: deleted, _rows_count = collector.delete()
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py", line 435, in delete
Jul 19 18:17:50 updates pulpcore-worker-6: signals.post_delete.send(
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 180, in send
Jul 19 18:17:50 updates pulpcore-worker-6: return [
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
Jul 19 18:17:50 updates pulpcore-worker-6: (receiver, receiver(signal=self, sender=sender, **named))
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py", line 338, in cleanup_subrepos
Jul 19 18:17:50 updates pulpcore-worker-6: subrepo = instance.repository
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
Jul 19 18:17:50 updates pulpcore-worker-6: rel_obj = self.get_object(instance)
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
Jul 19 18:17:50 updates pulpcore-worker-6: return qs.get(self.field.get_reverse_related_filter(instance))
Jul 19 18:17:50 updates pulpcore-worker-6: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
Jul 19 18:17:50 updates pulpcore-worker-6: raise self.model.DoesNotExist(
Jul 19 18:17:51 updates pulpcore-api: pulp [4fd22c711ce248d1a5348a0e393a4f8f]:  - - [19/Jul/2022:17:17:51 +0000] "GET /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/ HTTP/1.1" 200 3335 "-" "OpenAPI-Generator/3.16.7/ruby"

paused

$ hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused" '
[
  {
    "ID": "9057c237-d82c-410b-af81-701f917b7e16",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-07-19 17:16:58 UTC",
    "Ended at": null,
    "Duration": "00:01:41.365808",
    "Owner": "foreman_admin",
    "Task errors": [
      "Repository matching query does not exist."
    ]
  }
]

pulp href

# http_pulp /pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 943
Content-Type: application/json
Correlation-ID: 4ca178df822b496a96a30bc5c3864909
Date: Tue, 19 Jul 2022 17:20:25 GMT
Keep-Alive: timeout=15, max=100
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 updates.example.com
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Repository matching query does not exist.",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 380, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py\", line 66, in orphan_cleanup\n    c.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 340, in cleanup_subrepos\n    Variant.objects.filter(repository=subrepo).delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 338, in cleanup_subrepos\n    subrepo = instance.repository\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 187, in __get__\n    rel_obj = self.get_object(instance)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 154, in get_object\n    return qs.get(self.field.get_reverse_related_filter(instance))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 435, in get\n    raise self.model.DoesNotExist(\n"
    },
    "finished_at": "2022-07-19T17:17:50.928953Z",
    "logging_cid": "29a4aa8b574f485ebe72425ebb6505c5",
    "name": "pulpcore.app.tasks.orphan.orphan_cleanup",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "clean-up.content",
            "done": 2000,
            "message": "Clean up orphan Content",
            "state": "running",
            "suffix": null,
            "total": 15379
        }
    ],
    "pulp_created": "2022-07-19T17:17:40.032014Z",
    "pulp_href": "/pulp/api/v3/tasks/18ddd0f4-eb2c-4bd1-82b1-5125bd049f95/",
    "reserved_resources_record": [],
    "started_at": "2022-07-19T17:17:40.292569Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/38481210-0d06-4c37-96b2-20d50e237b6a/"
}

iballou · July 20, 2022, 4:23pm

I have a feeling this may be a new Pulp issue. Would you mind filing an issue here (Issues · pulp/pulpcore · GitHub) with the backtrace and the task information?

mindo · July 21, 2022, 5:19pm

Because we weren’t on the latest version we upgraded to foreman-3.3.0 and katello-4.5.0 but we still got the same error.
So we opened an issue upstream: https://github.com/pulp/pulpcore/issues/2996

dralley · August 4, 2022, 5:12am

I was able to reproduce and fix this, patch incoming.

gvde · August 29, 2022, 8:20am

This morning the same issue popped on my production server, more specifically on my content proxy. When running remove_orphan manually, the task for the main server runs fine but the one for the content proxy throws the error. I have the same setup as describe in the original post. I have force cancelled the paused task and started again: no change.

The installed version includes https://github.com/pulp/pulpcore/pull/3031 on both main server and content proxy, thus for me the patch does not fix the issue.

gvde · August 29, 2022, 8:30am

O.K. Scratch that. I have missed the follow-up pull request for pulp_rpm https://github.com/pulp/pulp_rpm/pull/2734

trendyandy · November 15, 2022, 7:09am

I have similar issue after upgrading my OS from RHEL7 to RHEL8. Orphan cleanup worked well on REHL7. However on RHEL8 the task gets always to paused state. It seems to stop when removing containers. I do not even have container support enabled in Katello.

Current versions:

foreman-3.3.1
katello-4.5.1
rubygem-foreman-tasks-6.0.2
python39-pulp-certguard-1.5.2-3
python39-pulp-cli-0.14.0-4
python39-pulp-rpm-3.18.7
python39-pulpcore-3.18.10

Logs:

root@foreman01 ~log # grep 23d2b202a2f940bcb90e20851fc33ff5 messages
Nov 14 15:11:19 foreman01 pulpcore-api[448113]: pulp [23d2b202a2f940bcb90e20851fc33ff5]: django.request:WARNING: Not Found: /pulp/api/v3/repositories/container/container/
Nov 14 15:11:19 foreman01 pulpcore-api[448113]: pulp [23d2b202a2f940bcb90e20851fc33ff5]:  - - [14/Nov/2022:14:11:19 +0000] "GET /pulp/api/v3/repositories/container/container/?limit=2000&offset=0 HTTP/1.1" 404 179 "-" "OpenAPI-Generator/2.10.3/ruby"

gvde · November 15, 2022, 7:11am

This is in Remove orphan fails

sajha · November 15, 2022, 1:57pm

We recently pushed a fix for this on nightly. Bug #35657: Remove orphan fails - Katello - Foreman

This will make its way into 4.6.1 and 4.7.0.