PulpDebClient::ApiError - HTTP 500 when running katello:delete_orphaned_content

Support
,
1

Problem:
Getting an error when trying to run foreman-rake katello:delete_orphaned_content

Expected outcome:
Orphan content to be removed

Foreman and Proxy versions:

# rpm -q foreman katello pulp python3-pulpcore python3-pulp-deb
foreman-2.3.5-1.el7.noarch
katello-3.18.5-1.el7.noarch
package pulp is not installed
python3-pulpcore-3.7.6-1.el7.noarch
python3-pulp-deb-2.9.2-1.el7.noarch

Distribution and version:
CentOS 7.9

Other relevant data:

image
image1123×893 31.1 KB

Full error:

 7: Actions::Pulp3::OrphanCleanup::DeleteOrphanRepositoryVersions (error) [ 9.31s / 9.31s ] Skip

Queue: default

Started at: 2021-09-27 14:23:35 UTC

Ended at: 2021-09-27 14:23:44 UTC

Real time: 9.31s

Execution time (excluding suspended state): 9.31s

Input:

---
smart_proxy_id: 1
remote_user: admin
remote_cp_user: admin
current_request_id: 
current_timezone: UTC
current_user_id: 1
current_organization_id: 
current_location_id: 

Output:

--- {}

Error:

PulpDebClient::ApiError

Error message: the server returns an error HTTP status code: 500 Response headers: {"date"=>"Mon, 27 Sep 2021 14:23:44 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"text/html; charset=UTF-8", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"27", "vary"=>"Cookie", "via"=>"1.1 foreman-01.lab.local", "connection"=>"close"} Response body: <h1>Server Error (500)</h1>

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_deb_client-2.9.1/lib/pulp_deb_client/api_client.rb:81:in
  `call_api'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_deb_client-2.9.1/lib/pulp_deb_client/api/repositories_deb_versions_api.rb:196:in
  `list_with_http_info'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_deb_client-2.9.1/lib/pulp_deb_client/api/repositories_deb_versions_api.rb:109:in
  `list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:154:in
  `block in versions_list_for_repository'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:199:in
  `block in fetch_from_list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:194:in
  `loop'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:194:in
  `fetch_from_list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:154:in
  `versions_list_for_repository'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:147:in
  `block in repository_versions'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:146:in
  `collect'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/api/core.rb:146:in
  `repository_versions'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/smart_proxy_repository.rb:59:in
  `block in orphan_repository_versions'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/smart_proxy_repository.rb:57:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/smart_proxy_repository.rb:57:in
  `orphan_repository_versions'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/smart_proxy_repository.rb:39:in
  `delete_orphan_repository_versions'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp3/orphan_cleanup/delete_orphan_repository_versions.rb:10:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in
  `wrap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in
  `with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in
  `restore_current_request_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in
  `restore_curent_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in
  `block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in
  `block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in
  `block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in
  `with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in
  `execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:68:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in
  `run_user_code'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in
  `execute_job'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in
  `block (2 levels) in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in
  `block in invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in
  `invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in
  `block in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in
  `block (6 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in
  `local'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in
  `block (5 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in
  `block in <module:Sidekiq>'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in
  `block (4 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in
  `stats'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in
  `block (3 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in
  `block (2 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in
  `global'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in
  `block in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in
  `with_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in
  `with_job_hash_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in
  `dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in
  `process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in
  `process_one'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in
  `watchdog'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in
  `block in safe_thread'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in
  `block in create_with_logging_context'"
2

Bump. Any ideas?

3

Aparently the line that throws the error is trying to retirive a list of repo versions from Pulp:

In order to say anything, we need more information why Pulp returns 500. Is it always the same repository? Is there something odd/special about the repo? Does the Pulp log contain any additional information? Do we know the exact API call that fails?

4

yes, i would check the content of:

journalctl -u pulpcore-api
and see if you can spot the traceback

5

Traceback:

Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/deb/apt/b577a080-96b8-457c-a5d7-077aab88aba1/versions/
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: Traceback (most recent call last):
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: response = get_response(request)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: response = self.process_exception_by_middleware(e, request)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: return view_func(*args, **kwargs)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 114, in view
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: return self.dispatch(request, *args, **kwargs)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 505, in dispatch
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: response = self.handle_exception(exc)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 465, in handle_exception
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: self.raise_uncaught_exception(exc)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: raise exc
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 502, in dispatch
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: response = handler(request, *args, **kwargs)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/mixins.py", line 43, in list
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: return self.get_paginated_response(serializer.data)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 760, in data
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: ret = super().data
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 260, in data
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: self._data = self.to_representation(self.instance)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 678, in to_representation
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: self.child.to_representation(item) for item in iterable
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 678, in <listcomp>
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: self.child.to_representation(item) for item in iterable
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 529, in to_representation
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: ret[field.field_name] = field.to_representation(attribute)
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/repository.py", line 217, in to_representation
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: item_dict = {"count": count_detail.count, "href": count_detail.content_href}
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 857, in content_href
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: ctype_model = Content.objects.filter(pulp_type=self.content_type).first().cast().__class__
Sep 30 00:00:23 foreman-01.lab.local pulpcore-api[17769]: AttributeError: 'NoneType' object has no attribute 'cast'
6

Looks like that query returns empty (None). The value of pulp_type=self.content_type is deb.generic

7

The affected repo appears to be focal-updates. Not sure if that matters

8

i’ve opened this pulp issue that covers this, as this seems like a pulp bug: Issue #9481: AttributeError: 'NoneType' object has no attribute 'cast' thrown when listing repostiory versions - Pulp

1 Like
9

Just updated to katello 4.2, and I see that the patch is not going to land for long time.

Applied the changes manually with the following patch:

--- /usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py  2021-12-15 19:47:21.557677732 +0000
+++ ./repository-fixed.py       2021-12-15 19:41:23.959648140 +0000
@@ -1013,18 +1013,22 @@
     @property
     def content_href(self):
         """
-        Generate URLs for the content types present in the RepositoryVersion.
-
-        For each content type present in the RepositoryVersion, create the URL of the viewset of
-        that variety of content along with a query parameter which filters it by presence in this
-        RepositoryVersion.
+        Generate URLs for the content types added, removed, or present in the RepositoryVersion.
+        For each content type present in or removed from this RepositoryVersion, create the URL of
+        the viewset of that variety of content along with a query parameter which filters it by
+        presence in this RepositoryVersion summary.

         Args:
             obj (pulpcore.app.models.RepositoryVersion): The RepositoryVersion being serialized.
         Returns:
             dict: {<pulp_type>: <url>}
         """
-        ctype_model = Content.objects.filter(pulp_type=self.content_type).first().cast().__class__
+        repository = self.repository_version.repository.cast()
+        repository_content = RepositoryContent.objects.filter(repository=repository)
+        ctype_query = Content.objects.filter(
+            pulp_type=self.content_type, pk__in=repository_content.values_list("content", flat=True)
+        )
+        ctype_model = ctype_query.first().cast().__class__
         ctype_view = get_view_name_for_model(ctype_model, "list")
         try:
             ctype_url = reverse(ctype_view)
@@ -1032,7 +1036,7 @@
             # We've hit a content type for which there is no viewset.
             # There's nothing we can do here, except to skip it.
             return
-        repository = self.repository_version.repository.cast()
+
         repository_view = get_view_name_for_model(repository.__class__, "list")

         repository_url = reverse(repository_view)

Then applied it and restarted all services

patch /usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py  < pulpdeb_api_error500.patch

However, I am still getting same error when trying to run katello:delete_orphaned_content. Here is the stacktrace:

Dec 15 19:52:32 host pulpcore-api[11021]: pulp [23e6e70d41cb407b9f4a28b6429db7f2]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/deb/apt/93fe3eca-961c-4688-8782-9239955f1ef5/
Dec 15 19:52:32 host pulpcore-api[11021]: Traceback (most recent call last):
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
Dec 15 19:52:32 host pulpcore-api[11021]: response = get_response(request)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
Dec 15 19:52:32 host pulpcore-api[11021]: response = self.process_exception_by_middleware(e, request)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
Dec 15 19:52:32 host pulpcore-api[11021]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Dec 15 19:52:32 host pulpcore-api[11021]: return view_func(*args, **kwargs)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 125, in view
Dec 15 19:52:32 host pulpcore-api[11021]: return self.dispatch(request, *args, **kwargs)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 509, in dispatch
Dec 15 19:52:32 host pulpcore-api[11021]: response = self.handle_exception(exc)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 469, in handle_exception
Dec 15 19:52:32 host pulpcore-api[11021]: self.raise_uncaught_exception(exc)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Dec 15 19:52:32 host pulpcore-api[11021]: raise exc
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 506, in dispatch
Dec 15 19:52:32 host pulpcore-api[11021]: response = handler(request, *args, **kwargs)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/mixins.py", line 43, in list
Dec 15 19:52:32 host pulpcore-api[11021]: return self.get_paginated_response(serializer.data)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 745, in data
Dec 15 19:52:32 host pulpcore-api[11021]: ret = super().data
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 246, in data
Dec 15 19:52:32 host pulpcore-api[11021]: self._data = self.to_representation(self.instance)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 664, in to_representation
Dec 15 19:52:32 host pulpcore-api[11021]: self.child.to_representation(item) for item in iterable
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 664, in <listcomp>
Dec 15 19:52:32 host pulpcore-api[11021]: self.child.to_representation(item) for item in iterable
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 515, in to_representation
Dec 15 19:52:32 host pulpcore-api[11021]: ret[field.field_name] = field.to_representation(attribute)
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/repository.py", line 360, in to_representation
Dec 15 19:52:32 host pulpcore-api[11021]: item_dict = {"count": count_detail.count, "href": count_detail.content_href}
Dec 15 19:52:32 host pulpcore-api[11021]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 1031, in content_href
Dec 15 19:52:32 host pulpcore-api[11021]: ctype_model = ctype_query.first().cast().__class__
Dec 15 19:52:32 host pulpcore-api[11021]: AttributeError: 'NoneType' object has no attribute 'cast'
10

Added some logging statements for the variables around error, here is the output:

Dec 16 15:08:51 host pulpcore-api[4282]: pulp [c14f5cb5c1b444e38203ec458b6072cc]: pulpcore.app.models.repository:ERROR: self: <RepositoryVersionContentDetails: RepositoryVersionContentDetails object (2161)>
Dec 16 15:08:51 host pulpcore-api[4282]: pulp [c14f5cb5c1b444e38203ec458b6072cc]: pulpcore.app.models.repository:ERROR: repository: <AptRepository: focal-security-37252>
Dec 16 15:08:51 host pulpcore-api[4282]: pulp [c14f5cb5c1b444e38203ec458b6072cc]: pulpcore.app.models.repository:ERROR: repository_content: <QuerySet [<RepositoryContent: pk=2cf7cdc4-b62d-4799-9139-82350e7ca20c>, <RepositoryContent: pk=d095f450-78a3-432a-a887-d31d86215881>, <RepositoryContent: pk=691e8efd-1bd0-4b95-b721-d73a248cb45a>, <RepositoryContent: pk=7b38cd4e-7afd-4c3f-8eb4-a1cf726abcfb>, <RepositoryContent: pk=a00d5d1d-28c4-4d1e-8f85-4c7386d7759c>, <RepositoryContent: pk=41e73436-6b8e-46b4-8a3a-f4a355a6456d>, <RepositoryContent: pk=4335d8cb-2460-4db2-b41a-c3a89ffcd982>, <RepositoryContent: pk=6efc54af-ea3a-4fee-ae4b-af6c4e5a32f6>, <RepositoryContent: pk=ac4fa904-a05f-41e3-a388-f461824310c6>, <RepositoryContent: pk=28b3c338-cee1-4bac-b308-9a4b9656152a>, <RepositoryContent: pk=b9872f47-69b7-4f65-ba52-227ed48623c6>, <RepositoryContent: pk=741ce22f-6c61-457f-886b-ad274e20eeda>, <RepositoryContent: pk=b49a93f8-9876-4fe6-a400-9187136ffe5a>, <RepositoryContent: pk=eda0959a-9663-4014-9f7b-982de33e7864>, <RepositoryContent: pk=98a34fc1-7b42-455e-8409-ed98504b6075>, <RepositoryContent: pk=587405a6-790e-4500-82d1-f7c3596cc080>, <RepositoryContent: pk=2d8dc2bf-9657-4997-b32a-29b7b3f8d119>, <RepositoryContent: pk=89fcdf11-d5dd-41c6-bd08-80dd94e17883>, <RepositoryContent: pk=0c23eb58-9ab0-4360-a5af-cca1675b2d96>, <RepositoryContent: pk=0082107c-88fb-4744-b804-c67d276716e3>, '...(remaining elements truncated)...']>
Dec 16 15:08:51 host pulpcore-api[4282]: pulp [c14f5cb5c1b444e38203ec458b6072cc]: pulpcore.app.models.repository:ERROR: ctype_query : 'SELECT "core_content"."pulp_id", "core_content"."pulp_created", "core_content"."pulp_last_updated", "core_content"."pulp_type", "core_content"."upstream_id", "core_content"."timestamp_of_interest" FROM "core_content" WHERE ("core_content"."pulp_id" IN (SELECT U0."content_id" FROM "core_repositorycontent" U0 WHERE U0."repository_id" = 93fe3eca-961c-4688-8782-9239955f1ef5) AND "core_content"."pulp_type" = deb.generic)'
Dec 16 15:08:51 host pulpcore-api[4282]: pulp [c14f5cb5c1b444e38203ec458b6072cc]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/deb/apt/93fe3eca-961c-4688-8782-9239955f1ef5/versions/
Dec 16 15:08:51 host pulpcore-api[4282]: Traceback (most recent call last):
...
Dec 16 15:08:51 host pulpcore-api[4282]: AttributeError: 'NoneType' object has no attribute 'cast'
11

I think I figured out the problem correctly this time. Here’s the PR: https://github.com/pulp/pulpcore/pull/1779. The problem occurs when a model isn’t updated properly upon repository version deletion, so after applying the patch you need to either 1. delete the problematic repositories and recreate them or 2. create new repository versions for each problematic repository and then delete all previous repository versions.

12

Hello!
Is there a way to determine, which repositories are problematic ?

13

If you have the Pulp CLI installed and pointed at Katello, you can do pulp repository list to get a list of each repository. Then for each repository you can run pulp show --href {REPO_HREF}/versions/ and the problematic ones will return a 500 error. I’m a Pulp developer so I don’t know how to use Katello, so maybe a Katello dev can clarify the equivalent Katello calls.

@rgp Also, I’ve updated the patch to be a one line insertion.

14

There doesn’t seem to be deb plugin for pulp cli to remove the repos. There were 2 showing 500. I deleted them from katello, and then all repos looked good. After some messing about, 3 other repos started showing 500. I didn’t even touch or sync them. Strange. Deleted and recreated them too, and now I can run katello:delete_orphaned_content successfully. I’m running with the patch applied, and hopefully no more broken repos.

Here is the updated patch file:

--- /usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py  2022-01-04 16:05:40.773359945 +0000
+++ repository-fixed.py 2022-01-04 16:05:21.138101758 +0000
@@ -878,6 +878,9 @@
         repo_relations.filter(version_added=self).update(version_added=next_version)
         repo_relations.filter(version_removed=self).update(version_removed=next_version)

+        # Update next version's counts as they have been modified
+        next_version._compute_counts()
+
     def delete(self, **kwargs):
         """
         Deletes a RepositoryVersion
@@ -1013,18 +1016,22 @@
     @property
     def content_href(self):
         """
-        Generate URLs for the content types present in the RepositoryVersion.
-
-        For each content type present in the RepositoryVersion, create the URL of the viewset of
-        that variety of content along with a query parameter which filters it by presence in this
-        RepositoryVersion.
+        Generate URLs for the content types added, removed, or present in the RepositoryVersion.
+        For each content type present in or removed from this RepositoryVersion, create the URL of
+        the viewset of that variety of content along with a query parameter which filters it by
+        presence in this RepositoryVersion summary.

         Args:
             obj (pulpcore.app.models.RepositoryVersion): The RepositoryVersion being serialized.
         Returns:
             dict: {<pulp_type>: <url>}
         """
-        ctype_model = Content.objects.filter(pulp_type=self.content_type).first().cast().__class__
+        repository = self.repository_version.repository.cast()
+        repository_content = RepositoryContent.objects.filter(repository=repository)
+        ctype_query = Content.objects.filter(
+            pulp_type=self.content_type, pk__in=repository_content.values_list("content", flat=True)
+        )
+        ctype_model = ctype_query.first().cast().__class__
         ctype_view = get_view_name_for_model(ctype_model, "list")
         try:
             ctype_url = reverse(ctype_view)
@@ -1032,7 +1039,7 @@
             # We've hit a content type for which there is no viewset.
             # There's nothing we can do here, except to skip it.
             return
-        repository = self.repository_version.repository.cast()
+
         repository_view = get_view_name_for_model(repository.__class__, "list")

         repository_url = reverse(repository_view)
1 Like
15

Just FYI: pulp-cli-deb now has some functionality. It has been packaged for inclusion with Katello 4.3 (yum install tfm-pulpcore-python3-pulp-cli-deb on el7).

I also have an open PR to package it for Katello 4.1 and 4.2 as well, but that has yet to be merged.