Uunable to sync redhat repos and refresh manifest warning

Hello
Problem:
It is unable to sync redhat repos. We have this error in gui is :

Error message: the server returns an error
HTTP status code: 500
Response headers: {"date"=>"Sat, 26 Mar 2022 12:53:45 GMT", "server"=>"gunicorn", "content-type"=>"text/html; charset=UTF-8", "x-frame-options"=>"DENY", "content-length"=>"145", "vary"=>"Cookie", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"< a correlation id>", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 <katelo server hostname>", "connection"=>"close"}
Response body: 
<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

when we try to refresh redhat manifest, we have a warning with the same issue.
And error in /var/log/messages is :

pulpcore-api: pulp [fbe75f2a-446f-44f8-8d2f-05d8d4454769]: django.request:ERROR: Internal Server Error: /pulp/api/v3/remotes/rpm/rpm/be4bfe1d-a80e-436b-bbce-8e28a7fdc78f/

when we try to get this, we have this issue :

#curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://<katello server hostname>/pulp/api/v3/remotes/rpm/rpm/be4bfe1d-a80e-436b-bbce-8e28a7fdc78f/

<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

pulp2 was migrated to pulp3 before upgrade in katello 4.x

Foreman and Proxy versions:
Foreman 3.1
katello 4.3
Distribution and version:
Redhat 7.9

Check the production.log. I guess you have this problem: PulpRpmClient::ApiError HTTP 500 during sync repository

Thank you.
I have this error in production.log when I refresh manifest

2022-03-27T11:31:18 [E|bac|6f6785fd] Error message: the server returns an error
 6f6785fd | HTTP status code: 500
 6f6785fd | Response headers: {"date"=>"Sun, 27 Mar 2022 09:31:18 GMT", "server"=>"gunicorn", "content-type"=>"text/html; charset=UTF-8", "x-frame-options"=>"DENY", "content-length"=>"145", "vary"=>"Cookie", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"<correlation_id>", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 <katello server>", "connection"=>"close"}
 6f6785fd | Response body:
 6f6785fd | <!doctype html>
 6f6785fd | <html lang="en">
 6f6785fd | <head>
 6f6785fd |   <title>Server Error (500)</title>
 6f6785fd | </head>
 6f6785fd | <body>
 6f6785fd |   <h1>Server Error (500)</h1><p></p>
 6f6785fd | </body>
 6f6785fd | </html>
 6f6785fd |  (PulpRpmClient::ApiError)
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/remotes_rpm_api.rb:318:in `partial_update_with_http_info'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/remotes_rpm_api.rb:262:in `partial_update'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository.rb:150:in `remote_partial_update'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository.rb:138:in `update_remote'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository.rb:188:in `refresh_if_needed'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/orchestration/repository/refresh_if_needed.rb:12:in `invoke_external_task'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:19:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 6f6785fd | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
6f6785fd | [ sidekiq ]
 6f6785fd | [ concurrent-ruby ]

and production.log when I sync a repo

2022-03-27T11:45:09 [I|app|f220ea19] Processing by Katello::SyncManagementController#sync_status as JSON
2022-03-27T11:45:09 [I|app|f220ea19]   Parameters: {"repoids"=>["12479"], "organization_id"=>"1", "_"=>"1648374291322"}
2022-03-27T11:45:09 [I|app|f220ea19] Completed 200 OK in 48ms (Views: 0.1ms | ActiveRecord: 9.7ms | Allocations: 23823)
2022-03-27T11:45:09 [E|bac|b8960c17] Error in progress calculation
2022-03-27T11:45:09 [E|bac|b8960c17] private method `select' called for nil:NilClass (NoMethodError)
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/repository/presenters/abstract_sync_presenter.rb:37:in `sync_task'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:37:in `total_units'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:7:in `progress'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/repository/sync.rb:35:in `run_progress'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:18:in `block in run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:34:in `with_progress_calculation'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
  b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 b8960c17 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 b8960c17 | [ sidekiq ]
 b8960c17 | [ concurrent-ruby ]

It looks like some repos were not properly migrated to pulp3. But there were no errors during migration.

The first error log looks a lot like the issue I have mentioned. It cannot update the remote because encryption of some fields is incorrect. You could check in the database. If you have more than 100 remotes in the database chance it some repos are broken until you ran the repair script.

thank you for your help.
The repair script is this one : Fix a migration mistake resulting in improperly encrypted remote fields by dralley · Pull Request #2333 · pulp/pulpcore · GitHub ?
Is it compatible with pulpcore 3.16 ?
thanks

I would highly recommend to verify this issue in the database, before trying the repair script. I would also recommend to wait until the repair script gets officially release with the next katello version. Otherwise, be sure to have a backup or snaphot to revert to before trying the repair script…

Thank you very much. I’m not sure i want to take the risk :sweat_smile:
I have found a method but that implies deleting all versions of CV and CCV where rhel repos in error are added.
It’s not a problem in my case.
It is easier to fix this issue if foreman katello configuration is automated.

Execute manifest refresh
For example :

hammer subscription refresh-manifest --organization <organization name>

Get pulp remote_href list in /var/log/messages

cat /var/log/messages |grep pulpcore |grep -i  "django.request:ERROR: Internal Server Error:" \
  | cut -d":" -f8 |sort -u |grep -v sync |sed "s/^ //g"

Extract repo list

https://<katello server>/katello/api/repositories?per_page=1000

→ With this extract, it is possible to get repo names from remote_href (it is probably better to script this step)

Remove each repo in all CV that use them
Publish new version of CV
Publish new version and promote to all lifecycle environments of CCV
Remove all old versions of CV and CCV where repo is added

Disable Redhat repos (Content > Redhat Repositories)
→ the tasks go in pending state with error. For each task, go in dynflow console, skip the first step. resume task.
Re-enable Redhat repos (Content > Redhat Repositories)
Synchronize Redhat repos
re-add repo to CV
Publish new version of CV
Publish new version and promote to all lifecycle environments of all CCV

The repair script has been merged with the pulp upstream packages and is already available for the current Katello release. Running “yum update” will include the new pulp version, at least it did in my case.

I haven’t seen a release announcement. It looks to me as if there are only some new pulpcore rpms since yesterday, but no updates to katello nor docs including repair instructions out there, yet.

I am sure they will arrive in the Katello docs with the next update, as this is an important upgrade issue and can hurt a few people, especially with smart proxies.

Here is the underlying pulp issue, including instructions on how to run the command

Thank you.
What will happen to smart proxies ? and in which version ?

Any content proxy upgraded from 4.2 to 4.3 using the buggy pulpcore version has basically the same problem, requiring to run the repair script after upgrading to the fixed version.

Deleting all affected repositories and recreating them works, too, but can be difficult if you have many repositories showing this problem. If you start a manual sync of all repositories on your katello server any sync failing with error 500 and the messages as shown above has this issue.

And as you have noticed, you’ll have to completely remove the repository from the system including from all content views in all existing versions. And depending on your internet connection it may take a while until all the repositories are fully synced again.

For the content proxy, you could remove all lifecycle environments and then add them again.

The repair script will fix the problem in a second…

1 Like

As stated in the pulp issue I linked, you need to run the repair script on the foreman server and ALL proxies. As soon as you updated to the pulp-core release mentioned there, you will have the script.

1 Like

LOL I’ve done this quite a number of times in the past. It takes forever and a day. Just glad I don’t pay internet by traffic …

Thank you all for your help