Update Counts task finishes with warning result

Support
,
1

Problem: Update task finishes with warning result all the time

Foreman and Proxy versions: 3.18.1/4.20.1

Foreman and Proxy plugin versions:

Distribution and version:RHEL 9.8

Other relevant data:

During weekend I updated Foreman to 3.18.1/4.20.1 and by chance I noticed that Update Count tasks have been finishing almost always with warning result on all of my Smart Proxies probably since upgrade to 3.17.

The error is:

 5013b3f2 | HTTP status code: 502
 5013b3f2 | Response headers: {"date"=>"Sat, 27 Jun 2026 23:16:42 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
 5013b3f2 | Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 5013b3f2 | <html><head>
 5013b3f2 | <title>502 Proxy Error</title>
 5013b3f2 | </head><body>
 5013b3f2 | <h1>Proxy Error</h1>
 5013b3f2 | <p>The proxy server received an invalid
 5013b3f2 | response from an upstream server.<br />
 5013b3f2 | The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
 5013b3f2 | </body></html>
 5013b3f2 |  (PulpRpmClient::ApiError)
2

Can you find the dynflow task output for the step that fails on the task. Also, what is the smart proxy version for this? Are the foreman and proxy upgraded to the same versions?

3

Hello, sorry for the late reply, I was on holiday. The smart proxy and foreman server are on the same versions, which is 4.20.1 currently. I am not expert in dynflow console, under the Run tab I found the

Actions::Katello::CapsuleContent::UpdateContentCounts task with foolowing error:

---
- "/usr/share/gems/gems/pulp_rpm_client-3.32.9/lib/pulp_rpm_client/api_client.rb:71:in
  `call_api'"
- "/usr/share/gems/gems/pulp_rpm_client-3.32.9/lib/pulp_rpm_client/api/content_packages_api.rb:303:in
  `list_with_http_info'"
- "/usr/share/gems/gems/pulp_rpm_client-3.32.9/lib/pulp_rpm_client/api/content_packages_api.rb:164:in
  `list'"
- "/usr/share/gems/gems/katello-4.20.1/app/services/katello/pulp3/repository_mirror.rb:210:in
  `count_by_pulpcore_type'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:216:in
  `block in translate_counts'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:213:in
  `each'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:213:in
  `translate_counts'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:198:in
  `process_repository'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:181:in
  `block in repos_content_count'"
- "/usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation/delegation.rb:88:in
  `each'"
- "/usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation/delegation.rb:88:in
  `each'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:180:in
  `repos_content_count'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:238:in
  `global_content_counts'"
- "/usr/share/gems/gems/katello-4.20.1/app/models/katello/concerns/smart_proxy_extensions.rb:159:in
  `update_content_counts!'"
- "/usr/share/gems/gems/katello-4.20.1/app/lib/actions/katello/capsule_content/update_content_counts.rb:19:in
  `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:590:in `block (3 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/usr/share/gems/gems/activesupport-7.0.10/lib/active_support/execution_wrapper.rb:92:in
  `wrap'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action/progress.rb:15:in `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:33:in `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:28:in `pass'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware.rb:20:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/usr/share/gems/gems/foreman-tasks-11.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/stack.rb:24:in `call'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/middleware/world.rb:31:in `execute'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:589:in `block (2 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:588:in `catch'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:588:in `block in execute_run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:491:in `block in with_error_handling'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:491:in `catch'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:491:in `with_error_handling'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:583:in `execute_run'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/action.rb:304:in `execute'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in
  `with_meta_calculation'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/director.rb:70:in `execute'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in
  `block (2 levels) in perform'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/executors.rb:18:in `run_user_code'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in
  `block in perform'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in
  `with_telemetry'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `perform'"
- "/usr/share/gems/gems/dynflow-2.0.0/lib/dynflow/executors/sidekiq/serialization.rb:28:in
  `perform'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels)
  in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'"
- "/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block
  in create_with_logging_context'"
4

That call_api hands it off to pulp so at this point, we’d be looking for logs in /var/log/messages for this call for more details..

5

I can see WORKER TIMEOUT in the /var/log/messages on the Smart Proxy. There are no error messages on the Foreman server.

Jul 13 06:03:26 smartproxy pulpcore-api[1249]: [2026-07-13 04:03:26 +0000] [1249] [CRITICAL] WORKER TIMEOUT (pid:2495853)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: [2026-07-13 04:03:27 +0000] [2495853] [ERROR] Error handling request /pulp/api/v3/content/rpm/packages/?fields=count&limit=1&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F019df1c2-7280-76cd-aad1-1547a83b4ad9%2Fversions%2F1%2F
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: Traceback (most recent call last):
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 134, in handle
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.handle_request(listener, req, client, addr)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 177, in handle_request
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     respiter = self.wsgi(environ, resp.start_response)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/wsgi.py", line 124, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/base.py", line 140, in get_response
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = self._middleware_chain(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django_guid/middleware.py", line 79, in middleware
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = response or self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/whitenoise/middleware.py", line 123, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = response or self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = response or self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = response or self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = response or self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = response or self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = response or self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/pulpcore/middleware.py", line 38, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/pulpcore/middleware.py", line 170, in __call__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return self.get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = get_response(request)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return view_func(*args, **kwargs)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/rest_framework/viewsets.py", line 125, in view
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return self.dispatch(request, *args, **kwargs)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/rest_framework/views.py", line 512, in dispatch
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     response = handler(request, *args, **kwargs)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/rest_framework/mixins.py", line 40, in list
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     page = self.paginate_queryset(queryset)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/rest_framework/generics.py", line 175, in paginate_queryset
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return self.paginator.paginate_queryset(queryset, self.request, view=self)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/rest_framework/pagination.py", line 401, in paginate_queryset
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return list(queryset[self.offset:self.offset + self.limit])
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 400, in __iter__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self._fetch_all()
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 1886, in _fetch_all
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self._result_cache = list(self._iterable_class(self))
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 93, in __iter__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     results = compiler.execute_sql(
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:               ^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     cursor.execute(sql, params)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/backends/utils.py", line 67, in execute
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return self._execute_with_wrappers(
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return executor(sql, params, many, context)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/backends/utils.py", line 89, in _execute
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return self.cursor.execute(sql, params)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/cursor.py", line 93, in execute
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self._conn.wait(
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/connection.py", line 453, in wait
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return waiting.wait(gen, self.pgconn.socket, interval=interval)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "psycopg_c/_psycopg/waiting.pyx", line 201, in psycopg_c._psycopg.wait_c
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     sys.exit(1)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: SystemExit: 1
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: pulp [-]:  - - [13/Jul/2026:04:03:27 +0000] "GET /pulp/api/v3/content/rpm/packages/?fields=count&limit=1&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F019df1c2-7280-76cd-aad1-1547a83b4ad9%2Fversions%2F1%2F HTTP/1.1" 500 0 "-" "-"
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: [2026-07-13 04:03:27 +0000] [2495853] [ERROR] Exception in worker process
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: Traceback (most recent call last):
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 375, in _set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.connection.autocommit = autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/_connection_base.py", line 182, in autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self._set_autocommit(value)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/connection.py", line 466, in _set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.set_autocommit(value)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/connection.py", line 471, in set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.wait(self._set_autocommit_gen(value))
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/connection.py", line 453, in wait
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return waiting.wait(gen, self.pgconn.socket, interval=interval)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "psycopg_c/_psycopg/waiting.pyx", line 198, in psycopg_c._psycopg.wait_c
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/_connection_base.py", line 188, in _set_autocommit_gen
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     yield from self._check_intrans_gen("autocommit")
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/_connection_base.py", line 260, in _check_intrans_gen
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     raise e.ProgrammingError(
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: psycopg.ProgrammingError: can't change 'autocommit' now: connection in transaction status ACTIVE
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: The above exception was the direct cause of the following exception:
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: Traceback (most recent call last):
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/gunicorn/arbiter.py", line 608, in spawn_worker
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     worker.init_process()
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/pulpcore/app/entrypoint.py", line 79, in init_process
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     super().init_process()
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/gunicorn/workers/base.py", line 143, in init_process
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.run()
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/pulpcore/app/entrypoint.py", line 87, in run
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.app_status.delete()
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/pulpcore/app/models/status.py", line 123, in delete
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self._old_status.delete(*args, **kwargs)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib64/python3.12/contextlib.py", line 80, in inner
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     with self._recreate_cm():
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:          ^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/transaction.py", line 216, in __enter__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     connection.set_autocommit(
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 496, in set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self._set_autocommit(autocommit)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 374, in _set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     with self.wrap_database_errors:
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:          ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 375, in _set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.connection.autocommit = autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/_connection_base.py", line 182, in autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self._set_autocommit(value)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/connection.py", line 466, in _set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.set_autocommit(value)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/connection.py", line 471, in set_autocommit
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     self.wait(self._set_autocommit_gen(value))
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/connection.py", line 453, in wait
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     return waiting.wait(gen, self.pgconn.socket, interval=interval)
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "psycopg_c/_psycopg/waiting.pyx", line 198, in psycopg_c._psycopg.wait_c
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/_connection_base.py", line 188, in _set_autocommit_gen
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     yield from self._check_intrans_gen("autocommit")
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:   File "/usr/lib/python3.12/site-packages/psycopg/_connection_base.py", line 260, in _check_intrans_gen
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]:     raise e.ProgrammingError(
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: django.db.utils.ProgrammingError: can't change 'autocommit' now: connection in transaction status ACTIVE
Jul 13 06:03:27 smartproxy pulpcore-api[2495853]: [2026-07-13 04:03:27 +0000] [2495853] [INFO] Worker exiting (pid: 2495853)
6

That suggests we are hitting a slow pulp query on some large repo..I can raise a bug for you around this. For reference, what is the repository with the largest content count you’re syncing to the smart proxy?

7

Increasing timeouts on the smart proxy for pulp workers could be a fix in the meantime to allow the long running query to finish before the worker timeouts. For ex, increase to 180 seconds, twice the default 90 seconds.

foreman-installer
–foreman-proxy-content-pulpcore-api-service-worker-timeout 180
–foreman-proxy-content-pulpcore-content-service-worker-timeout 180

8

Well, thank you for the workaround I am going to test it. How do I find the repository with largest content count ?

9

So I set this parameter on 3 of our smartproxies and the rest of them left unchanged. The Update Count task fails on all of them but the TIMEOUT is reported only on those unchanged. The TIMEOUT error also always appears several minutes after the Update task fails so may be it is not strongly related to it.

I have also noticed following warnings when starting proxy service on the smartproxies with modified parameter:

Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243075]: [2026-07-16 17:29:12 +0000] [3243075] [INFO] Booting worker with pid: 3243075
Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243075]: pulp [None]: pulpcore.app.entrypoint:WARNING: API_APP_TTL (120) is smaller than double the gunicorn timeout (90.0). You may experience workers wrongly reporting as missing
Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243077]: [2026-07-16 17:29:12 +0000] [3243077] [INFO] Booting worker with pid: 3243077
Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243077]: pulp [None]: pulpcore.app.entrypoint:WARNING: API_APP_TTL (120) is smaller than double the gunicorn timeout (90.0). You may experience workers wrongly reporting as missing
Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243080]: [2026-07-16 17:29:12 +0000] [3243080] [INFO] Booting worker with pid: 3243080
Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243080]: pulp [None]: pulpcore.app.entrypoint:WARNING: API_APP_TTL (120) is smaller than double the gunicorn timeout (90.0). You may experience workers wrongly reporting as missing
Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243082]: [2026-07-16 17:29:12 +0000] [3243082] [INFO] Booting worker with pid: 3243082
Jul 16 19:29:12 smartproxy.ux.mbid.cz pulpcore-api[3243082]: pulp [None]: pulpcore.app.entrypoint:WARNING: API_APP_TTL (120) is smaller than double the gunicorn timeout (90.0). You may experience workers wrongly reporting as missing
Jul 16 19:29:13 smartproxy.ux.mbid.cz pulpcore-api[3243088]: [2026-07-16 17:29:13 +0000] [3243088] [INFO] Booting worker with pid: 3243088
Jul 16 19:29:13 smartproxy.ux.mbid.cz pulpcore-api[3243088]: pulp [None]: pulpcore.app.entrypoint:WARNING: API_APP_TTL (120) is smaller than double the gunicorn timeout (90.0). You may experience workers wrongly reporting as missing
10

Is that warning on smart proxies with only one timeout increased or both? Not sure how long that query runs and what the appropriate timeout setting could be that is not excessive..We’ll have to dig into that in postgresql.

Mostly wanted to get a sense of the size of the environment. So things like count of rpms in your larger repos, number of content views, ball park estimate of how many versions your CVs have.. The long running query filters all rpms by repository versions(CV version)..A ballpark estimate of how many rpms and CV versions we are dealing with will help to file an optimization request in pulp..

If you want something specific, something like this in foreman-rake console will get you the rpm count of largest repo you have.:

Katello::RepositoryRpm.count will tell us how many Repo version - RPM records the query is working on.

Katello::RepositoryRpm.group(:repository_id).order('count_all desc').limit(1).count.values.first

Katello::ContentViewVersion.count will show how many repo versions you have on smart proxy assuming everything is being synced to all smart proxies.

11

Hello,

the warning appears only on smatproxies with timeout increased.

irb(main):001:0> Katello::RepositoryRpm.count
=> 18776062
irb(main):002:0> Katello::RepositoryRpm.group(:repository_id).order('count_all desc').limit(1).count.values.first
=> 105231
irb(main):003:0> Katello::ContentViewVersion.count
=> 84