502 Proxy Error during Pulp3 Migration

am123 · March 10, 2022, 7:54am

Problem:
Upon running
foreman-maintain content prepare
the migration always shows the below error:

Error message: the server returns an error
HTTP status code: 502
Response headers: {"date"=>"Wed, 09 Mar 2022 16:03:34 GMT", "server"=>"Apache", "content-length"=>"429", "connection"=>"close", "content-type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/pulp/api/v3/pulp2content/">GET&nbsp;/pulp/api/v3/pulp2content/</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

I have tried multiple times to run the command but with the same outcome

Expected outcome:
“foreman-maintain content prepare” works without errors

Foreman and Proxy versions:
foreman-2.3.5
foreman-proxy-2.3.5
katello-3.18.5

Foreman and Proxy plugin versions:
foreman-tasks 3.0.6
foreman_ansible 6.1.1
foreman_bootdisk 17.0.2
foreman_default_hostgroup 6.0.0
foreman_discovery 16.3.4
foreman_docker 5.0.0
foreman_remote_execution 4.2.2
katello 3.18.5

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:

/var/log/foreman-proxy/proxy.log did not contain any relevant logs
The only thing I could find in /var/log/foreman/production.log is the following part with “CONTENT_MIGRATION”. This is around the time the migration failed

2022-03-09T17:03:26 [I|app|0563781c] Processing by UsersController#login as HTML
2022-03-09T17:03:26 [I|app|0563781c]   Rendering users/login.html.erb within layouts/login
2022-03-09T17:03:26 [I|app|0563781c]   Rendered common/_login.html.erb (Duration: 3.4ms | Allocations: 6169)
2022-03-09T17:03:26 [I|app|0563781c]   Rendered users/login.html.erb within layouts/login (Duration: 10.9ms | Allocations: 12396)
2022-03-09T17:03:26 [I|app|0563781c]   Rendering layouts/base.html.erb
2022-03-09T17:03:26 [I|app|0563781c]   Rendered layouts/base.html.erb (Duration: 4.9ms | Allocations: 7416)
2022-03-09T17:03:26 [I|app|0563781c] Completed 200 OK in 26ms (Views: 20.2ms | ActiveRecord: 1.9ms | Allocations: 27641)
2022-03-09T17:03:33 [I|app|] CONTENT_MIGRATION - Importing Repository type=yum duration=50736.8
2022-03-09T17:03:34 [I|app|] CONTENT_MIGRATION - Importing Content type=rpm duration=818.22
2022-03-09T17:03:34 [I|app|] CONTENT_MIGRATION - Importing Content type=modulemd duration=3.61
2022-03-09T17:03:54 [I|app|eee781c3] Started GET "/rhsm/" for 10.116.1.221 at 2022-03-09 17:03:54 +0100
2022-03-09T17:03:54 [I|app|eee781c3] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2022-03-09T17:03:54 [I|app|eee781c3]   Parameters: {"root"=>{}}
2022-03-09T17:03:54 [I|app|eee781c3]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2022-03-09T17:03:54 [I|app|eee781c3]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (Duration: 9.2ms | Allocations: 4134)
2022-03-09T17:03:54 [I|app|eee781c3] Completed 200 OK in 50ms (Views: 14.7ms | ActiveRecord: 6.1ms | Allocations: 15645)
2022-03-09T17:03:54 [I|app|7e3ca0bd] Started GET "/rhsm/consumers/e2b1c031-942c-4406-9508-0de9ee8c38cf/certificates/serials" for 10.116.1.221 at 2022-03-09 17:03:54 +0100

I also got this from the dynflow console

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_2to3_migration_client-0.11.0/lib/pulp_2to3_migration_client/api_client.rb:81:in
  `call_api'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_2to3_migration_client-0.11.0/lib/pulp_2to3_migration_client/api/pulp2_content_api.rb:139:in
  `list_with_http_info'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_2to3_migration_client-0.11.0/lib/pulp_2to3_migration_client/api/pulp2_content_api.rb:43:in
  `list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:279:in
  `operate_on_errata'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:295:in
  `import_errata'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:341:in
  `import_content_type'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:116:in
  `block (4 levels) in import_pulp3_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/katello/logging.rb:6:in
  `time'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:115:in
  `block (3 levels) in import_pulp3_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:114:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:114:in
  `block (2 levels) in import_pulp3_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:109:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:109:in
  `block in import_pulp3_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/katello/logging.rb:6:in
  `time'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp3/migration.rb:108:in
  `import_pulp3_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp3/import_migration.rb:11:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in
  `wrap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in
  `with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in
  `restore_current_request_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in
  `restore_curent_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in
  `block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in
  `block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in
  `block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in
  `with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in
  `execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:68:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in
  `run_user_code'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in
  `execute_job'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in
  `block (2 levels) in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in
  `block in invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in
  `invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in
  `block in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in
  `block (6 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in
  `local'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in
  `block (5 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in
  `block in <module:Sidekiq>'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in
  `block (4 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in
  `stats'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in
  `block (3 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in
  `block (2 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in
  `global'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in
  `block in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in
  `with_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in
  `with_job_hash_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in
  `dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in
  `process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in
  `process_one'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in
  `watchdog'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in
  `block in safe_thread'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in
  `block in create_with_logging_context'"

Thank you in advance!

am123 · March 10, 2022, 8:21am

This is what I found in /var/log/httpd/foreman-ssl_error_ssl.log:

[Wed Mar 09 17:04:05.362921 2022] [proxy_http:error] [pid 12384] (20014)Internal error: [client 10.116.6.145:54570] AH01102: error reading status line from remote server httpd-UDS:0
[Wed Mar 09 17:04:05.363034 2022] [proxy:error] [pid 12384] [client 10.116.6.145:54570] AH00898: Error reading from remote server returned by /pulp/api/v3/pulp2content/

/var/log/httpd/foreman-ssl_access_ssl.log

10.116.6.145 - admin [09/Mar/2022:17:03:34 +0100] "GET /pulp/api/v3/pulp2content/?limit=2000&offset=0&pulp2_content_type_id=erratum&pulp2_last_updated__gt=0 HTTP/1.1" 502 429 "-" "OpenAPI-Generator/0.11.0/ruby"

I’m not sure if this is realted, but here are the journal logs:

Mär 09 17:04:05 prod-satellite-001.cp.wien pulpcore-api[2625]: [2022-03-09 17:04:05 +0100] [2625] [CRITICAL] WORKER TIMEOUT (pid:16990)
Mär 09 17:04:05 prod-satellite-001.cp.wien pulpcore-api[2625]: [2022-03-09 16:04:05 +0000] [16990] [INFO] Worker exiting (pid: 16990)
Mär 09 17:04:06 prod-satellite-001.cp.wien pulpcore-api[2625]: [2022-03-09 17:04:06 +0100] [26065] [INFO] Booting worker with pid: 26065

am123 · March 10, 2022, 9:24am

This is the output of the “foreman-maintain content prepare” command that I ran again today.

Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Checking for valid Katello configuraton.
Starting task.
2022-03-10 08:56:16 +0100: Content migration starting. These steps may take a while to complete. Refer to `foreman-maintain                                                                                                                           2022-03-10 08:56:26 +0100: Content migration starting. These steps may take a while to complete. Refer to `foreman-maintain                                                                                                                           2022-03-10 09:22:30 +0100: Migrating rpm content to Pulp 3 10099/3097130971
2022-03-10 09:59:55 +0100: Importing migrated content type rpm: 16/16Migration failed, You will want to investigate: https://prod-satellite-001.cp.wien/foreman_tasks/tasks/2d2d59ac-989e-4aab-83db-3f8111a97e44
rake aborted!
ForemanTasks::TaskError: Task 2d2d59ac-989e-4aab-83db-3f8111a97e44: Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 502
Response headers: {"date"=>"Thu, 10 Mar 2022 08:59:30 GMT", "server"=>"Apache", "content-length"=>"429", "connection"=>"close", "content-type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/pulp/api/v3/pulp2content/">GET&nbsp;/pulp/api/v3/pulp2content/</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/lib/katello/tasks/pulp3_migration.rake:41:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

am123 · March 17, 2022, 11:13am

Does someone have an idea what could cause this?

The alternative would be to resetup katello fully, but I would like to avoid this if possible.

Thank!

quba42 · March 17, 2022, 12:02pm

I am going to ping @Justin_Sherrill one more time here, since he might have some more ideas. (Please note that he is on US EST time, so he probably won’t respond before later in the afternoon).

rednight08 · July 21, 2023, 10:55am

Is the above error resolved?
Can anyone update me with the cause behind this error and with a solution?