PulpAnsibleClient::ApiError when syncing ansible collections

Support
1

Problem:

I just upgraded from Foreman v3.13 / Katello v4.15 to Foreman v3.14 / Katello v4.16 and experience the following Issue when trying to sync ansible collections from https://galaxy.ansible.com:

2025-04-20T14:40:24 [I|app|74f1585b] CONTENT_INDEX type=Katello::AnsibleCollection success=false duration=60110.79
2025-04-20T14:40:24 [E|bac|74f1585b] Error message: the server returns an error
 74f1585b | HTTP status code: 502
 74f1585b | Response headers: {"date"=>"Sun, 20 Apr 2025 12:39:24 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
 74f1585b | Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 74f1585b | <html><head>
 74f1585b | <title>502 Proxy Error</title>
 74f1585b | </head><body>
 74f1585b | <h1>Proxy Error</h1>
 74f1585b | <p>The proxy server received an invalid
 74f1585b | response from an upstream server.<br />
 74f1585b | The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
 74f1585b | </body></html>
 74f1585b |  (PulpAnsibleClient::ApiError)
 74f1585b | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api_client.rb:83:in `call_api'
 74f1585b | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:239:in `list_with_http_info'
 74f1585b | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:155:in `list'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:155:in `content_unit_list'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:168:in `fetch_content_list'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:136:in `block (2 levels) in pulp_units_batch_for_repo'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `loop'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `block in pulp_units_batch_for_repo'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `each'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `import_all'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1033:in `block (2 levels) in index_content'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/lib/katello/logging.rb:8:in `time'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1032:in `block in index_content'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `each'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `index_content'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/lib/actions/katello/repository/index_content.rb:22:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 74f1585b | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in `wrap'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/world.rb:31:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `catch'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `block in execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `catch'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:583:in `execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:304:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/director.rb:70:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors.rb:18:in `run_user_code'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 74f1585b | [ sidekiq ]
 74f1585b | [ concurrent-ruby ]
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: , execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: pending
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: planning
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: planned
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: running
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: stopped  result: success
2025-04-20T14:40:25 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::Sync, id: fa8f64ee-ec02-47e2-b7f6-07c54392d2be, execution_plan_id: 5e613d90-f3a1-4bd0-96f0-4943710e40f9} state changed: stopped  result: warning

The task stops with state == ‘warning’ and the following message:

Error message: the server returns an error
HTTP status code: 502
Response headers: {"date"=>"Sun, 20 Apr 2025 14:30:42 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

For testing purposes, I was using the following requirements.yml:

---
collections:
  - name: ansible.posix
  - name: ansible.utils
  - name: community.general
...

Expected outcome:

The sync runs successfully.

Foreman and Proxy versions:

# rpm -qa | grep -iIE '^foreman-[0-9]\.[0-9]|^katello-[0-9]\.[0-9]'
foreman-3.14.0-1.el9.noarch
katello-4.16.0-1.el9.noarch

Foreman and Proxy plugin versions:

# rpm -qa | grep -iIE '^rubygem-pulp_ansible'
rubygem-pulp_ansible_client-0.22.3-1.el9.noarch

Distribution and version:

# cat /etc/redhat-release
AlmaLinux release 9.5 (Teal Serval)

Other relevant data:

As mentioned above, I’m trying to sync a few collections from https://galaxy.ansible.com. I’m not using a proxy in this environment, nor did I set up any authentication towards the upstream repo.
If any more information is required, please let me know.

Thanks :slight_smile:

2

Hi @adrian ,

The 502 proxy error is suspicious, it sounds network related. I just tried creating an Ansible repo with your requirements.txt and it worked fine for me. Specifically:

---
collections:
  - name: ansible.posix
  - name: ansible.utils
  - name: community.general

and nothing else.

On your Foreman server, can you try a simple network test, like:

curl "https://galaxy.ansible.com/api/v3/plugin/ansible/search/collection-versions/?dependency=community.network&offset=0&limit=10"

Does that work without issues?

We may need to take this to the Pulp community if we get stuck, since the network error seems to be originating in Pulp.

3

Hi @iballou

thanks for your answer. I’ve just tried the curl command and it seems to work fine. I get a fairly large json-blob as return.

I’m currently running the sync again, just to make sure it wasn’t a temporary network issue.
I’ll update the post once the task has finished.

4

The sync task just finished with a warning state. The stack trace seems to be identical:

2025-04-21T18:29:02 [E|bac|a37fb09d] Error message: the server returns an error
 a37fb09d | HTTP status code: 502
 a37fb09d | Response headers: {"date"=>"Mon, 21 Apr 2025 16:28:02 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
 a37fb09d | Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 a37fb09d | <html><head>
 a37fb09d | <title>502 Proxy Error</title>
 a37fb09d | </head><body>
 a37fb09d | <h1>Proxy Error</h1>
 a37fb09d | <p>The proxy server received an invalid
 a37fb09d | response from an upstream server.<br />
 a37fb09d | The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
 a37fb09d | </body></html>
 a37fb09d |  (PulpAnsibleClient::ApiError)
 a37fb09d | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api_client.rb:83:in `call_api'
 a37fb09d | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:239:in `list_with_http_info'
 a37fb09d | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:155:in `list'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:155:in `content_unit_list'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:168:in `fetch_content_list'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:136:in `block (2 levels) in pulp_units_batch_for_repo'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `loop'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `block in pulp_units_batch_for_repo'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `each'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `import_all'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1033:in `block (2 levels) in index_content'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/lib/katello/logging.rb:8:in `time'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1032:in `block in index_content'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `each'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `index_content'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/lib/actions/katello/repository/index_content.rb:22:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 a37fb09d | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in `wrap'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/world.rb:31:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `catch'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `block in execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `catch'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:583:in `execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:304:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/director.rb:70:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors.rb:18:in `run_user_code'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 a37fb09d | [ sidekiq ]
 a37fb09d | [ concurrent-ruby ]
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: , execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: pending
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: planning
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: planned
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: running
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: stopped  result: success
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::Sync, id: 0e810148-97fc-4693-9fb2-24940af1bf8b, execution_plan_id: 56262bde-b62a-498a-98ac-9e1ff3b3d723} state changed: stopped  result: warning
5

Wonder if the “HTTP Proxy Policy” on the repository is set to anything?

6

There’s no proxy set on the Repository:
image

7

It might be helpful to look at the Pulp remote - that dictates how the repository is synced by Pulp.

You can find these with sudo pulp ansible remote list

If you’re able to share the remote configuration, I can compare it to the working remote on my system.

8

Sure, here’s the output of the command:

# pulp ansible remote list
zsh: correct 'ansible' to '.ansible' [nyae]? n
[
  {
    "pulp_href": "/pulp/api/v3/remotes/ansible/collection/01965310-8895-792b-89c8-4a2002ea5776/",
    "prn": "prn:ansible.collectionremote:01965310-8895-792b-89c8-4a2002ea5776",
    "pulp_created": "2025-04-20T11:58:34.648588Z",
    "pulp_last_updated": "2025-04-30T18:49:33.946498Z",
    "name": "collections-45285082",
    "url": "https://galaxy.ansible.com/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": true,
    "proxy_url": null,
    "pulp_labels": {},
    "download_concurrency": null,
    "max_retries": null,
    "policy": "immediate",
    "total_timeout": 3600.0,
    "connect_timeout": 60.0,
    "sock_connect_timeout": 60.0,
    "sock_read_timeout": 600.0,
    "headers": null,
    "rate_limit": 0,
    "hidden_fields": [
      {
        "name": "client_key",
        "is_set": false
      },
      {
        "name": "proxy_username",
        "is_set": false
      },
      {
        "name": "proxy_password",
        "is_set": false
      },
      {
        "name": "username",
        "is_set": false
      },
      {
        "name": "password",
        "is_set": false
      },
      {
        "name": "token",
        "is_set": false
      }
    ],
    "requirements_file": "---\ncollections:\n  - name: ansible.posix\n  - name: ansible.utils\n  - name: community.general\n...",
    "auth_url": null,
    "sync_dependencies": true,
    "signed_only": false,
    "last_sync_task": {
      "pk": "019687fd-2f62-7a41-99d3-0bd8326d5a03",
      "error": null,
      "state": "completed",
      "finished_at": "2025-04-30T18:47:18.264856+00:00",
      "pulp_created": "2025-04-30T18:37:19.075528+00:00"
    }
  }
]

Let me know if there’s anything obvious.

I’ve ran a sync about one hour ago and it synced successfully once. However, I’m currently getting the dreaded:

Error message: the server returns an error
HTTP status code: 502
Response headers: {"date"=>"Wed, 30 Apr 2025 18:47:44 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

I’ll go ahead and test a few things, maybe I can get it working again.

9

Update: I can’t seem to replicate it anymore so the problem still persists.
What I’ve tried so far:

  • removing the repo + product
  • setting the ‘Orphaned Content Protection Time’ to 5 mins
  • wait 15 mins and run foreman-rake katello:delete_orphaned_content RAILS_ENV=production
  • recreate the product, repository and try to sync

I also restarted the server and reran foreman-installer --scenario katello after reboots just to make sure. The Issue is still the same.

I’ll try disabling the pulp plugin and enabling it later today using foreman-installer --scenario katello --foreman-proxy-content-enable-ansible false / true to check if that does anything.

10

Update - I got the sync working again.

Here’s what I did:

  • set ‘Orphaned Content Protection Time’ to 5 mins
  • remove all ansible repositories
  • my ansible content repos were in a separate product, so I removed that as well (might not apply to all
  • wait a few minutes (more then the time I set in ‘Orphaned Content Protection Time’ earlier)
  • run foreman-rake katello:delete_orphaned_content RAILS_ENV=production
  • check the WebUI for tasks to complete or observe tail -F /var/log/foreman/production.log
  • stop all services once the task is finished foreman-maintain service stop
  • run foreman-installer --scenario katello --foreman-proxy-content-enable-ansible false
  • check all foreman_proxy_content settings in /etc/foreman-installer/scenarios.d/katello-answers.yaml to make sure they’re correct. If not, adjust accordingly.
  • check the health using foreman-maintain service status and foreman-maintain health check
  • stop all running services using foreman-maintain service stop
  • run foreman-installer --scenario katello --foreman-proxy-content-enable-ansible true
  • double-check /etc/foreman-installer/scenarios.d/katello-answers.yaml as mentioned previously
  • recreate the product(s) and repository / repositories
  • sync

After completing all of these steps, the sync worked again for me although ymmv.

I’ll go ahead and mark this as the solution.

11

Huh, I’m not sure why the above steps would fix a proxy error, maybe some installer configuration was out of date which was causing Pulp troubles. Anyway, glad to see it’s solved.

12

Since this instance has seen upgrades from c7 → al8 → al9 over the years, I’d consider config drift the most likely. I think the key here was disabling and possibly removing the old config and then enabling the plugin with a, probably fresh, config again.