PulpAnsibleClient::ApiError when syncing ansible collections

adrian · April 21, 2025, 3:49pm

Problem:

I just upgraded from Foreman v3.13 / Katello v4.15 to Foreman v3.14 / Katello v4.16 and experience the following Issue when trying to sync ansible collections from https://galaxy.ansible.com:

2025-04-20T14:40:24 [I|app|74f1585b] CONTENT_INDEX type=Katello::AnsibleCollection success=false duration=60110.79
2025-04-20T14:40:24 [E|bac|74f1585b] Error message: the server returns an error
 74f1585b | HTTP status code: 502
 74f1585b | Response headers: {"date"=>"Sun, 20 Apr 2025 12:39:24 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
 74f1585b | Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 74f1585b | <html><head>
 74f1585b | <title>502 Proxy Error</title>
 74f1585b | </head><body>
 74f1585b | <h1>Proxy Error</h1>
 74f1585b | <p>The proxy server received an invalid
 74f1585b | response from an upstream server.<br />
 74f1585b | The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
 74f1585b | </body></html>
 74f1585b |  (PulpAnsibleClient::ApiError)
 74f1585b | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api_client.rb:83:in `call_api'
 74f1585b | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:239:in `list_with_http_info'
 74f1585b | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:155:in `list'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:155:in `content_unit_list'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:168:in `fetch_content_list'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:136:in `block (2 levels) in pulp_units_batch_for_repo'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `loop'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `block in pulp_units_batch_for_repo'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `each'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `import_all'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1033:in `block (2 levels) in index_content'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/lib/katello/logging.rb:8:in `time'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1032:in `block in index_content'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `each'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `index_content'
 74f1585b | /usr/share/gems/gems/katello-4.16.0/app/lib/actions/katello/repository/index_content.rb:22:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 74f1585b | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in `wrap'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 74f1585b | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/world.rb:31:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `catch'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `block in execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `catch'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:583:in `execute_run'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:304:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/director.rb:70:in `execute'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors.rb:18:in `run_user_code'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 74f1585b | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 74f1585b | [ sidekiq ]
 74f1585b | [ concurrent-ruby ]
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: , execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: pending
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: planning
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: planned
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: running
2025-04-20T14:40:24 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5e055a47-3164-4edd-ae54-9ffd5d613dca, execution_plan_id: 39c8b7eb-3bcd-4897-a138-09c5557ccc3e} state changed: stopped  result: success
2025-04-20T14:40:25 [I|bac|74f1585b] Task {label: Actions::Katello::Repository::Sync, id: fa8f64ee-ec02-47e2-b7f6-07c54392d2be, execution_plan_id: 5e613d90-f3a1-4bd0-96f0-4943710e40f9} state changed: stopped  result: warning

The task stops with state == ‘warning’ and the following message:

Error message: the server returns an error
HTTP status code: 502
Response headers: {"date"=>"Sun, 20 Apr 2025 14:30:42 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

For testing purposes, I was using the following requirements.yml:

---
collections:
  - name: ansible.posix
  - name: ansible.utils
  - name: community.general
...

Expected outcome:

The sync runs successfully.

Foreman and Proxy versions:

# rpm -qa | grep -iIE '^foreman-[0-9]\.[0-9]|^katello-[0-9]\.[0-9]'
foreman-3.14.0-1.el9.noarch
katello-4.16.0-1.el9.noarch

Foreman and Proxy plugin versions:

# rpm -qa | grep -iIE '^rubygem-pulp_ansible'
rubygem-pulp_ansible_client-0.22.3-1.el9.noarch

Distribution and version:

# cat /etc/redhat-release
AlmaLinux release 9.5 (Teal Serval)

Other relevant data:

As mentioned above, I’m trying to sync a few collections from https://galaxy.ansible.com. I’m not using a proxy in this environment, nor did I set up any authentication towards the upstream repo.
If any more information is required, please let me know.

Thanks

iballou · April 21, 2025, 4:10pm

Hi @adrian ,

The 502 proxy error is suspicious, it sounds network related. I just tried creating an Ansible repo with your requirements.txt and it worked fine for me. Specifically:

---
collections:
  - name: ansible.posix
  - name: ansible.utils
  - name: community.general

and nothing else.

On your Foreman server, can you try a simple network test, like:

curl "https://galaxy.ansible.com/api/v3/plugin/ansible/search/collection-versions/?dependency=community.network&offset=0&limit=10"

Does that work without issues?

We may need to take this to the Pulp community if we get stuck, since the network error seems to be originating in Pulp.

adrian · April 21, 2025, 4:20pm

Hi @iballou

thanks for your answer. I’ve just tried the curl command and it seems to work fine. I get a fairly large json-blob as return.

I’m currently running the sync again, just to make sure it wasn’t a temporary network issue.
I’ll update the post once the task has finished.

adrian · April 21, 2025, 4:35pm

The sync task just finished with a warning state. The stack trace seems to be identical:

2025-04-21T18:29:02 [E|bac|a37fb09d] Error message: the server returns an error
 a37fb09d | HTTP status code: 502
 a37fb09d | Response headers: {"date"=>"Mon, 21 Apr 2025 16:28:02 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
 a37fb09d | Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 a37fb09d | <html><head>
 a37fb09d | <title>502 Proxy Error</title>
 a37fb09d | </head><body>
 a37fb09d | <h1>Proxy Error</h1>
 a37fb09d | <p>The proxy server received an invalid
 a37fb09d | response from an upstream server.<br />
 a37fb09d | The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
 a37fb09d | </body></html>
 a37fb09d |  (PulpAnsibleClient::ApiError)
 a37fb09d | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api_client.rb:83:in `call_api'
 a37fb09d | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:239:in `list_with_http_info'
 a37fb09d | /usr/share/gems/gems/pulp_ansible_client-0.22.3/lib/pulp_ansible_client/api/content_collection_versions_api.rb:155:in `list'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:155:in `content_unit_list'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:168:in `fetch_content_list'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:136:in `block (2 levels) in pulp_units_batch_for_repo'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `loop'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/pulp3/pulp_content_unit.rb:127:in `block in pulp_units_batch_for_repo'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `each'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/services/katello/content_unit_indexer.rb:26:in `import_all'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1033:in `block (2 levels) in index_content'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/lib/katello/logging.rb:8:in `time'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1032:in `block in index_content'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `each'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/models/katello/repository.rb:1031:in `index_content'
 a37fb09d | /usr/share/gems/gems/katello-4.16.0/app/lib/actions/katello/repository/index_content.rb:22:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 a37fb09d | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in `wrap'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 a37fb09d | /usr/share/gems/gems/foreman-tasks-10.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/world.rb:31:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `catch'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `block in execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `catch'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:583:in `execute_run'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:304:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/director.rb:70:in `execute'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors.rb:18:in `run_user_code'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 a37fb09d | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 a37fb09d | [ sidekiq ]
 a37fb09d | [ concurrent-ruby ]
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: , execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: pending
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: planning
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: planned
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: running
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::CapsuleSync, id: 7d2cfb6d-3276-4ddf-b6fb-1b741c9698b4, execution_plan_id: 584c1a05-891e-4171-b1e0-48b178914b1c} state changed: stopped  result: success
2025-04-21T18:29:02 [I|bac|a37fb09d] Task {label: Actions::Katello::Repository::Sync, id: 0e810148-97fc-4693-9fb2-24940af1bf8b, execution_plan_id: 56262bde-b62a-498a-98ac-9e1ff3b3d723} state changed: stopped  result: warning

sajha · April 21, 2025, 5:37pm

Wonder if the “HTTP Proxy Policy” on the repository is set to anything?

adrian · April 21, 2025, 5:46pm

There’s no proxy set on the Repository:

iballou · April 30, 2025, 4:38pm

It might be helpful to look at the Pulp remote - that dictates how the repository is synced by Pulp.

You can find these with sudo pulp ansible remote list

If you’re able to share the remote configuration, I can compare it to the working remote on my system.

adrian · April 30, 2025, 6:52pm

Sure, here’s the output of the command:

# pulp ansible remote list
zsh: correct 'ansible' to '.ansible' [nyae]? n
[
  {
    "pulp_href": "/pulp/api/v3/remotes/ansible/collection/01965310-8895-792b-89c8-4a2002ea5776/",
    "prn": "prn:ansible.collectionremote:01965310-8895-792b-89c8-4a2002ea5776",
    "pulp_created": "2025-04-20T11:58:34.648588Z",
    "pulp_last_updated": "2025-04-30T18:49:33.946498Z",
    "name": "collections-45285082",
    "url": "https://galaxy.ansible.com/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": true,
    "proxy_url": null,
    "pulp_labels": {},
    "download_concurrency": null,
    "max_retries": null,
    "policy": "immediate",
    "total_timeout": 3600.0,
    "connect_timeout": 60.0,
    "sock_connect_timeout": 60.0,
    "sock_read_timeout": 600.0,
    "headers": null,
    "rate_limit": 0,
    "hidden_fields": [
      {
        "name": "client_key",
        "is_set": false
      },
      {
        "name": "proxy_username",
        "is_set": false
      },
      {
        "name": "proxy_password",
        "is_set": false
      },
      {
        "name": "username",
        "is_set": false
      },
      {
        "name": "password",
        "is_set": false
      },
      {
        "name": "token",
        "is_set": false
      }
    ],
    "requirements_file": "---\ncollections:\n  - name: ansible.posix\n  - name: ansible.utils\n  - name: community.general\n...",
    "auth_url": null,
    "sync_dependencies": true,
    "signed_only": false,
    "last_sync_task": {
      "pk": "019687fd-2f62-7a41-99d3-0bd8326d5a03",
      "error": null,
      "state": "completed",
      "finished_at": "2025-04-30T18:47:18.264856+00:00",
      "pulp_created": "2025-04-30T18:37:19.075528+00:00"
    }
  }
]

Let me know if there’s anything obvious.

I’ve ran a sync about one hour ago and it synced successfully once. However, I’m currently getting the dreaded:

Error message: the server returns an error
HTTP status code: 502
Response headers: {"date"=>"Wed, 30 Apr 2025 18:47:44 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

I’ll go ahead and test a few things, maybe I can get it working again.

adrian · May 4, 2025, 8:50am

Update: I can’t seem to replicate it anymore so the problem still persists.
What I’ve tried so far:

removing the repo + product
setting the ‘Orphaned Content Protection Time’ to 5 mins
wait 15 mins and run foreman-rake katello:delete_orphaned_content RAILS_ENV=production
recreate the product, repository and try to sync

I also restarted the server and reran foreman-installer --scenario katello after reboots just to make sure. The Issue is still the same.

I’ll try disabling the pulp plugin and enabling it later today using foreman-installer --scenario katello --foreman-proxy-content-enable-ansible false / true to check if that does anything.

adrian · May 4, 2025, 11:04am

Update - I got the sync working again.

Here’s what I did:

set ‘Orphaned Content Protection Time’ to 5 mins
remove all ansible repositories
my ansible content repos were in a separate product, so I removed that as well (might not apply to all
wait a few minutes (more then the time I set in ‘Orphaned Content Protection Time’ earlier)
run foreman-rake katello:delete_orphaned_content RAILS_ENV=production
check the WebUI for tasks to complete or observe tail -F /var/log/foreman/production.log
stop all services once the task is finished foreman-maintain service stop
run foreman-installer --scenario katello --foreman-proxy-content-enable-ansible false
check all foreman_proxy_content settings in /etc/foreman-installer/scenarios.d/katello-answers.yaml to make sure they’re correct. If not, adjust accordingly.
check the health using foreman-maintain service status and foreman-maintain health check
stop all running services using foreman-maintain service stop
run foreman-installer --scenario katello --foreman-proxy-content-enable-ansible true
double-check /etc/foreman-installer/scenarios.d/katello-answers.yaml as mentioned previously
recreate the product(s) and repository / repositories
sync

After completing all of these steps, the sync worked again for me although ymmv.

I’ll go ahead and mark this as the solution.

iballou · May 8, 2025, 8:39pm

Huh, I’m not sure why the above steps would fix a proxy error, maybe some installer configuration was out of date which was causing Pulp troubles. Anyway, glad to see it’s solved.

adrian · May 8, 2025, 8:52pm

Since this instance has seen upgrades from c7 → al8 → al9 over the years, I’d consider config drift the most likely. I think the key here was disabling and possibly removing the old config and then enabling the plugin with a, probably fresh, config again.