Delete orphaned content task failing: Server Error (500)

Support
1

Hi

Problem:
When trying to delete orphaned content of of the tasks fail and gets stuck in state paused

Foreman and Proxy versions:

foreman-3.3.0-1.el8
foreman-proxy-3.3.0-1.el8
katello-4.5.0-1.el8
python39-pulpcore-3.18.6-1.el8

Distribution and version:

AlmaLinux8.6 ( Upgraded from CentOS7 in place with a leapp )

Applied patch from Fixed orphan cleanup task failing with `Repository matching query does by ipanova · Pull Request #2734 · pulp/pulp_rpm · GitHub

Other relevant data:

foreman-rake katello:delete_orphaned_content RAILS_ENV=production

hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused" '
[
  {
    "ID": "3270eded-55b8-4ad1-8141-8b689e6abf1d",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-09-15 09:33:01 UTC",
    "Ended at": null,
    "Duration": "00:42:37.677505",
    "Owner": "foreman_admin",
    "Task errors": [
      "Error message: the server returns an error\nHTTP status code: 500\nResponse headers: {\"date\"=>\"Thu, 15 Sep 2022 09:33:06 GMT\", \"server\"=>\"gunicorn\", \"content-type\"=>\"text/html; charset=UTF-8\", \"x-frame-options\"=>\"DENY\", \"content-length\"=>\"145\", \"vary\"=>\"Cookie\", \"x-content-type-options\"=>\"nosniff\", \"referrer-policy\"=>\"same-origin\", \"correlation-id\"=>\"fe0bb7ba56f547c59144ac0258dde790\", \"access-control-expose-headers\"=>\"Correlation-ID\", \"via\"=>\"1.1 foremanserver.infra.intra\", \"connection\"=>\"close\"}\nResponse body: \n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>**Server Error (500)**</title>\n</head>\n<body>\n  <h1>Server Error (500)</h1><p></p>\n</body>\n</html>\n"
    ]
  }
]

Logs from /var/log/messages:

Sep 15 13:18:34 foremanserver pulpcore-api[16303]: pulp [74397de425cd4f0aad29935798a5518d]:  - - [15/Sep/2022:11:18:34 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.6p219"
Sep 15 13:18:34 foremanserver pulpcore-api[16303]: pulp [9182a33536a44229ad0f02ec2d19162b]:  - - [15/Sep/2022:11:18:34 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4119 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.6p219"
Sep 15 13:18:34 foremanserver pulpcore-api[16303]: pulp [af6c59fdda8e44f4bbb67ec3799ef532]:  - - [15/Sep/2022:11:18:34 +0000] "GET /pulp/api/v3/repositories/ansible/ansible/?limit=2000&offset=0 HTTP/1.1" 200 52 "-" "OpenAPI-Generator/0.13.1/ruby"
Sep 15 13:18:34 foremanserver pulpcore-api[16303]: pulp [b7c834fa9c0b476696a79df03fcf8134]:  - - [15/Sep/2022:11:18:34 +0000] "GET /pulp/api/v3/repositories/deb/apt/?limit=2000&offset=0 HTTP/1.1" 200 52 "-" "OpenAPI-Generator/2.18.0/ruby"
Sep 15 13:18:35 foremanserver pulpcore-api[16303]: pulp [408b0a0251944ec29e9bed88e0ae963b]:  - - [15/Sep/2022:11:18:35 +0000] "GET /pulp/api/v3/repositories/container/container/?limit=2000&offset=0 HTTP/1.1" 200 52 "-" "OpenAPI-Generator/2.10.3/ruby"
Sep 15 13:18:35 foremanserver pulpcore-api[16304]: pulp [b22f5bfe11ce430aa9707f3512c9c4aa]:  - - [15/Sep/2022:11:18:35 +0000] "GET /pulp/api/v3/repositories/file/file/?limit=2000&offset=0 HTTP/1.1" 200 52 "-" "OpenAPI-Generator/1.10.0/ruby"
Sep 15 13:18:35 foremanserver pulpcore-api[16301]: pulp [038378a711b848a19522c29ca9998b58]:  - - [15/Sep/2022:11:18:35 +0000] "GET /pulp/api/v3/repositories/python/python/?limit=2000&offset=0 HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.6.0/ruby"
Sep 15 13:18:35 foremanserver pulpcore-api[16301]: pulp [d37e2ed702134072a15c5ba819f223dd]:  - - [15/Sep/2022:11:18:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/?limit=2000&offset=0 HTTP/1.1" 200 112043 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:35 foremanserver pulpcore-api[16305]: pulp [9bb16a63dcd844eda46de5426eca3940]:  - - [15/Sep/2022:11:18:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/49ccd4ce-d476-44b1-865f-146d253ac73e/versions/?limit=2000&offset=0 HTTP/1.1" 200 370 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:35 foremanserver pulpcore-api[16305]: pulp [53bd1dc27af246938483f1a1abc02495]:  - - [15/Sep/2022:11:18:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/c986e28f-f58a-4895-8ed3-0a66bf4ca50f/versions/?limit=2000&offset=0 HTTP/1.1" 200 370 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:35 foremanserver pulpcore-api[16305]: pulp [7db2b3af90cf4c8eb25bff22d4bebcfb]:  - - [15/Sep/2022:11:18:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/845efe57-0615-4731-b53e-feca049b6239/versions/?limit=2000&offset=0 HTTP/1.1" 200 1049 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:36 foremanserver pulpcore-api[16302]: pulp [abe812bf53a54586bb4096469591e3fc]:  - - [15/Sep/2022:11:18:36 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/d1feef47-e7c7-4d61-9611-848fcc3c738c/versions/?limit=2000&offset=0 HTTP/1.1" 200 1039 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:36 foremanserver pulpcore-api[16302]: pulp [1c231e584fcb46e2b504aaa75f1b7a73]:  - - [15/Sep/2022:11:18:36 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/02194642-4aea-4c1e-bdaf-0e1ab67e5c5a/versions/?limit=2000&offset=0 HTTP/1.1" 200 1970 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:36 foremanserver pulpcore-api[16302]: pulp [85e273c9f03648f5bdd436dc4576d52f]:  - - [15/Sep/2022:11:18:36 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/60140c2f-b098-42f1-80bd-45fb8114d739/versions/?limit=2000&offset=0 HTTP/1.1" 200 1045 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:36 foremanserver pulpcore-api[16302]: pulp [1e6af6e4382145078c267cdd37293fe4]:  - - [15/Sep/2022:11:18:36 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/f5118149-14ac-4c21-9f1f-0b6c0ac2b5d5/versions/?limit=2000&offset=0 HTTP/1.1" 200 1045 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:37 foremanserver pulpcore-api[16302]: pulp [d40a637f37a84724884be765c319f615]:  - - [15/Sep/2022:11:18:37 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/651343fa-882a-45ae-82c5-712cd3190ba0/versions/?limit=2000&offset=0 HTTP/1.1" 200 24182 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:37 foremanserver pulpcore-api[16302]: pulp [2cb7fabed10847d99c2e0038f621c3cd]:  - - [15/Sep/2022:11:18:37 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/a4d5bb8e-df34-454d-b129-edb388875331/versions/?limit=2000&offset=0 HTTP/1.1" 200 2391 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:37 foremanserver pulpcore-api[16302]: pulp [064806f71dc64288b1583abfd58585df]:  - - [15/Sep/2022:11:18:37 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/ea154fd3-dacf-44f6-ba69-924af73bbcfa/versions/?limit=2000&offset=0 HTTP/1.1" 200 2193 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:38 foremanserver pulpcore-api[16303]: pulp [67d03e5774874eadb8a27ce419696488]:  - - [15/Sep/2022:11:18:38 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/54758c71-a839-4f81-97f3-fe249c139a4d/versions/?limit=2000&offset=0 HTTP/1.1" 200 6130 "-" "OpenAPI-Generator/3.17.4/ruby"
Sep 15 13:18:38 foremanserver pulpcore-api[16302]: pulp [40eb166c1c8147a5abf991298ee9bbe3]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/ee81ad25-4e0c-4c7f-9187-0d167ed3ec55/versions/
Sep 15 13:18:38 foremanserver pulpcore-api[16302]: Traceback (most recent call last):
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    response = get_response(request)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    return view_func(*args, **kwargs)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    return self.dispatch(request, *args, **kwargs)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    response = self.handle_exception(exc)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    response = self.handle_exception(exc)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    self.raise_uncaught_exception(exc)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    raise exc
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    response = handler(request, *args, **kwargs)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/mixins.py", line 43, in list
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    return self.get_paginated_response(serializer.data)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 768, in data
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    ret = super().data
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 253, in data
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    self._data = self.to_representation(self.instance)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 686, in to_representation
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    return [
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 687, in <listcomp>
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    self.child.to_representation(item) for item in iterable
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 522, in to_representation
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    ret[field.field_name] = field.to_representation(attribute)
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/pulpcore/app/serializers/repository.py", line 371, in to_representation
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    item_dict = {"count": count_detail.count, "href": count_detail.content_href}
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:  File "/usr/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 1118, in content_href
Sep 15 13:18:38 foremanserver pulpcore-api[16302]:    ctype_model = ctype_query.first().cast().__class__
Sep 15 13:18:38 foremanserver pulpcore-api[16302]: AttributeError: 'NoneType' object has no attribute 'cast'
Sep 15 13:18:38 foremanserver pulpcore-api[16302]: pulp [40eb166c1c8147a5abf991298ee9bbe3]:  - - [15/Sep/2022:11:18:38 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/ee81ad25-4e0c-4c7f-9187-0d167ed3ec55/versions/?limit=2000&offset=0 HTTP/1.1" 500 145 "-" "OpenAPI-Generator/3.17.4/ruby"

Thank you for help

3

I linked the wrong issue. But anyway, it’s separate enough to file separately. Could you file an issue upstream?

4

That looks exactly as the issue I had on older version with deb repositories. The patches are in katello 4.5. If you’ve upgraded from older version like 3.18 and only now are trying to use katello:delete_orphaned_content, then it may be the same issue.

To fix the repositories, I had to delete them and recreate them. Then katello:delete_orphaned_content started working without errors.

5

Thank you, i’ve found one repo with a 500 error:

 pulp show --href /pulp/api/v3/repositories/rpm/rpm/314c72b5-9d68-496c-afae-f75291744044/versions/
Error:
<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

with a pulp_href:

pulp show --href /pulp/api/v3/repositories/rpm/rpm/314c72b5-9d68-496c-afae-f75291744044/
{
  "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/314c72b5-9d68-496c-afae-f75291744044/",
  "pulp_created": "2021-05-27T21:30:26.951520Z",
  "versions_href": "/pulp/api/v3/repositories/rpm/rpm/314c72b5-9d68-496c-afae-f75291744044/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/314c72b5-9d68-496c-afae-f75291744044/versions/2/",
  **"name": "a4774596-5119-4227-b1f5-2085a9843055",**
**  "description": "a4774596-5119-4227-b1f5-2085a9843055",**
  "retain_repo_versions": null,
  "remote": null,
  "autopublish": false,
  "metadata_signing_service": null,
  "retain_package_versions": 0,
  "metadata_checksum_type": null,
  "package_checksum_type": null,
  "gpgcheck": 0,
  "repo_gpgcheck": 0,
  "sqlite_metadata": false
}

but the name doesn’t tell me nothing, “hammer repository list” doesn’t show it
How to know of witch repo(or ContentVIew?) this version is part of?

6

Could be a part of a content view and thus only uuid as name, not really sure. You could try to match one of the uuid to a repo from the katello api list. API documentation

7

Finally, i’ve found those repos, two by deduction method :slight_smile: ( two old RedHat fasttrack repos was empty, after delete them i 've got a next repo with error ) and with another one i’ve got a clear name: "“name”: “CentOS7_os_x86_64-4353736”
So i’ve delete two empty and CentOS7_os repos, then recreate latest one, add it to my CVs and orphan cleanup successfully terminated
Thank you for your help

1 Like
8

Hi,
I did what mentioned above:

  • deleted all content views with affected repositories and repositories
  • create the new ones
    but the problem persists.
    With
pulp rpm repository list

i have found that versions with affected repositories still exist in pulp, but I cannot find them via GUI or hammer. So, I cannot successfullu finish task:

foreman-rake katello:delete_orphaned_content

9

I’ve got the same issue

10

I’ve figured out how to fix it
/usr/lib/python3.9/site-packages/pulpcore/app/models/repository.py
around line 1118
try:
ctype_model = ctype_query.first().cast().class
except AttributeError:
#Bad repository with version issues
return

will allow it to continue and it fixes it and on the second run will probably delete it

11

I hit exacly the same error with foreman 3.3.1 and pulpcore 3.18.10.

12

Closing the loop, the cause seems to be this Pulp issue: RepositoryVersionContentDetails should not perfrom a content query when creating the content summary · Issue #3299 · pulp/pulpcore · GitHub

Which is fixed by this patch: Remove query from repository version content summary by gerrod3 · Pull Request #3300 · pulp/pulpcore · GitHub

Which is fixed in pulpcore 3.18.11 and later

1 Like
13

We just upgraded 3.2 to 3.3.1 and had this exact issue. pulpcore 3.18.12 was installed, so it may not be fully fixed by using 3.18.11+. We were able to quickly find the one with the issue by running the following single line and watching for a 500 response. We had quite a few, so we had to increase the limit to 300 with the --limit switch.

pulp rpm repository list --limit 300 | jq -rc '.[].versions_href' | xargs -d $'\n' -n 1 pulp show --href
1 Like