Import of most python content from Pypi is failing in Foreman 3.19.1/Katello 4.21

Just upgraded to Foreman 3.19.1/Katello 4.21 and noticed that my Python product that contain a repository syncing python content from Pypi is not working anymore. tired adding some different libraries to the Includes list and most if them fail with a similar error. Ex.

aiobotocore:
No declared artifact with relative path “aiobotocore-3.7.0-py3-none-any.whl.metadata” for content “<PythonPackageContent: aiobotocore [3.7.0] (bdist_wheel)>”

click:
No declared artifact with relative path “click-8.4.1-py3-none-any.whl.metadata” for content “<PythonPackageContent: click [8.4.1] (bdist_wheel)>”

ansible_pylibssh:
No declared artifact with relative path “ansible_pylibssh-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata” for content “<PythonPackageContent: ansible-pylibssh [1.3.0] (bdist_wheel)>”

One library that I did manage to sync is paramiko but most other seems to fail.

Some logs:

from journalctl:

journalctl -u 'pulpcore*' -S "2026-07-08 07:45:00" -o cat --no-pager
`
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:14 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 7696 "-" "rest-client/2.1.0 (linux x86_64) ruby/3.0.7p220"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Starting task id: 019f40b0-2a17-7c66-bce4-0d6948dcf007 in domain: default, task_type: pulpcore.app.tasks.base.ageneral_update, immediate: True, deferred: True
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Task completed 019f40b0-2a17-7c66-bce4-0d6948dcf007 in domain: default, task_type: pulpcore.app.tasks.base.ageneral_update, immediate: True, deferred: True, execution_time: 11786 μs
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:14 +0000] "PATCH /pulp/api/v3/remotes/python/python/019f40a6-8afa-7288-b28e-6e92fa417e3d/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:14 +0000] "GET /pulp/api/v3/tasks/019f40b0-2a17-7c66-bce4-0d6948dcf007/ HTTP/1.1" 200 1756 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:15 +0000] "POST /pulp/api/v3/repositories/python/python/019f40a6-8c3b-7d5d-9779-14ccb6e48aa8/sync/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Starting task id: 019f40b0-2aed-79f0-bd7e-9caf3fe6f34f in domain: default, task_type: pulp_python.app.tasks.sync.sync, immediate: False, deferred: True
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized project plugin allowlist_project, filtering ['ansible-pylibssh']
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized release plugin allowlist_release, filtering [<Requirement('ansible-pylibssh')>]
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized release plugin blocklist_release, filtering []
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized prerelease plugin with [re.compile('.+rc\\d+$'), re.compile('.+a(lpha)?\\d+$'), re.compile('.+b(eta)?\\d+$'), re.compile('.+dev\\d+$')]
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized prerelease plugin prerelease_release, filtering all packages
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.mirror:INFO: Syncing with https://pypi.org.
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.mirror:INFO: No metadata filters are enabled. Skipping metadata filtering
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.mirror:INFO: No release file filters are enabled. Skipping release file filtering
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:15 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1233 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.package:INFO: Fetching metadata for package: ansible-pylibssh (serial 0)
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:15 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1233 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:16 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1235 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:16 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1235 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:17 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1237 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:18 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1237 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Task[pulp_python.app.tasks.sync.sync] 019f40b0-2aed-79f0-bd7e-9caf3fe6f34f failed (ValueError: No declared artifact with relative path "ansible_pylibssh-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata" for content "<PythonPackageContent: ansible-pylibssh [1.3.0] (bdist_wheel)>") in domain: default
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO:   File "/usr/lib/python3.12/site-packages/pulpcore/tasking/tasks.py", line 73, in _execute_task
    result = task_function()
             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pulp_python/app/tasks/sync.py", line 60, in sync
    DeclarativeVersion(first_stage, repository, mirror).create()
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/declarative_version.py", line 163, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/api.py", line 217, in create_pipeline
    await asyncio.gather(*futures)
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/api.py", line 40, in __call__
    await self.run()
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 318, in run
    await self._handle_remote_artifacts(batch)
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 418, in _handle_remote_artifacts
    raise ValueError(
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 2648 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "GET /pulp/api/v3/distributions/python/pypi/?base_path=ssc%2FLibrary%2Fcustom%2Fpython%2Fpypi HTTP/1.1" 200 759 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "GET /pulp/api/v3/distributions/python/pypi/019f40a6-8f01-7e98-b61a-12155ea15a7f/ HTTP/1.1" 200 707 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "PATCH /pulp/api/v3/distributions/python/pypi/019f40a6-8f01-7e98-b61a-12155ea15a7f/ HTTP/1.1" 200 707 "-" "OpenAPI-Generator/3.27.2/ruby"

production log:

grep 7d229deb production.log
2026-07-08T07:45:14 [I|app|7d229deb] Started POST "/katello/api/v2/repositories/1303/sync?organization_id=1" for 10.107.17.18 at 2026-07-08 07:45:14 +0000
2026-07-08T07:45:14 [I|app|7d229deb] Processing by Katello::Api::V2::RepositoriesController#sync as HTML
2026-07-08T07:45:14 [I|app|7d229deb]   Parameters: {"id"=>"1303", "organization_id"=>"1", "api_version"=>"v2", "repository"=>{"id"=>"1303"}}
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: , execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: pending
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: planning
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: planned
2026-07-08T07:45:14 [I|app|7d229deb]   Rendered /usr/share/gems/gems/katello-4.21.0/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 18.1ms | Allocations: 14313)
2026-07-08T07:45:14 [I|app|7d229deb]   Rendered layout /usr/share/gems/gems/katello-4.21.0/app/views/katello/api/v2/layouts/resource.json.erb (Duration: 18.3ms | Allocations: 14406)
2026-07-08T07:45:14 [I|app|7d229deb] Completed 202 Accepted in 280ms (Views: 17.8ms | ActiveRecord: 26.2ms | Allocations: 98026)
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: running
2026-07-08T07:45:19 [E|bac|7d229deb] No declared artifact with relative path "ansible_pylibssh-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata" for content "<PythonPackageContent: ansible-pylibssh [1.3.0] (bdist_wheel)>" (Katello::Errors::Pulp3Error)
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:115:in `block in check_for_errors'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:113:in `each'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:113:in `check_for_errors'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:169:in `poll_external_task'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/polling.rb:22:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/cancellable.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:33:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 7d229deb | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/execution_wrapper.rb:92:in `wrap'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/progress.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:33:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:48:in `restore_current_user'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/world.rb:31:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:588:in `catch'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:588:in `block in execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `block in with_error_handling'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `catch'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `with_error_handling'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:583:in `execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:304:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/director.rb:95:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors.rb:18:in `run_user_code'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 7d229deb | [ sidekiq ]
 7d229deb | [ concurrent-ruby ]
2026-07-08T07:45:19 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: stopped  result: warning
2026-07-08T07:45:19 [I|app|7d229deb] ForemanWebhooks::EventSubscriber: actions.katello.repository.sync_failed.event.foreman event received

versions:

# rpm -qa | sort | egrep 'katello|foreman|pulpcore|pulp-python'
ansible-collection-theforeman-foreman-5.11.0-1.el9.noarch
ansiblerole-foreman_scap_client-0.5.0-1.el9.noarch
foreman-3.19.1-1.el9.noarch
foreman-cli-3.19.1-1.el9.noarch
foreman-dynflow-sidekiq-3.19.1-1.el9.noarch
foreman-installer-3.19.1-1.el9.noarch
foreman-installer-katello-3.19.1-1.el9.noarch
foreman-obsolete-packages-1.11-2.el9.noarch
foreman-postgresql-3.19.1-1.el9.noarch
foreman-proxy-3.19.1-1.el9.noarch
foreman-redis-3.19.1-1.el9.noarch
foreman-release-3.19.1-1.el9.noarch
foreman-selinux-3.19.1-1.el9.noarch
foreman-service-3.19.1-1.el9.noarch
katello-4.21.0-1.el9.noarch
katello-certs-tools-2.10.0-1.el9.noarch
katello-client-bootstrap-1.7.9-2.el9.noarch
katello-common-4.21.0-1.el9.noarch
katello-repos-4.21.0-1.el9.noarch
katello-selinux-5.2.0-1.el9.noarch
pulpcore-obsolete-packages-1.3.0-1.el9.noarch
pulpcore-selinux-2.2.0-1.el9.x86_64
python3.12-pulpcore-3.105.6-1.el9.noarch
python3.12-pulp-python-3.27.2-1.el9.noarch
rubygem-foreman_ansible-17.0.5-1.fm3_19.el9.noarch
rubygem-foreman_maintain-1.15.0-1.el9.noarch
rubygem-foreman_openscap-12.1.0-1.fm3_19.el9.noarch
rubygem-foreman_remote_execution-16.6.4-1.fm3_19.el9.noarch
rubygem-foreman-tasks-12.1.1-1.fm3_19.el9.noarch
rubygem-foreman_webhooks-5.0.3-1.fm3_19.el9.noarch
rubygem-hammer_cli_foreman-3.19.0-1.el9.noarch
rubygem-hammer_cli_foreman_ansible-0.8.2-1.fm3_18.el9.noarch
rubygem-hammer_cli_foreman_openscap-0.2.3-1.fm3_17.el9.noarch
rubygem-hammer_cli_foreman_remote_execution-0.4.2-1.fm3_19.el9.noarch
rubygem-hammer_cli_foreman_tasks-0.0.24-1.fm3_17.el9.noarch
rubygem-hammer_cli_katello-1.21.0-1.el9.noarch
rubygem-katello-4.21.0-1.el9.noarch
rubygem-pulpcore_client-3.105.5-1.el9.noarch

Not sure if this is the caused, but did you run a full sync for the python content as advised in Katello 4.21.0 is now available

Can also add that if I only add paramiko to the include list and the sync works, it is only available on the Foreman server and not on the proxies (I see proxy sync tasks run and complete OK after the sync) .
ex:

$ curl -s https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/

<html>
<head><title>Index of /pulp/content/xxx/Library/custom/python/pypi/</title></head>
<body bgcolor="white">
<h1>Index of /pulp/content/xxx/Library/custom/python/pypi/</h1>
<hr><pre><a href="../">../</a>
<a href="paramiko-4.0.0-py3-none-any.whl">paramiko-4.0.0-py3-none-any.whl</a>                                                                     08-Jul-2026 08:35  223.9 kB
<a href="paramiko-4.0.0-py3-none-any.whl.metadata">paramiko-4.0.0-py3-none-any.whl.metadata</a>                                                            08-Jul-2026 08:35
<a href="paramiko-4.0.0.tar.gz">paramiko-4.0.0.tar.gz</a>                                                                               08-Jul-2026 08:35  1.6 MB
<a href="paramiko-5.0.0-py3-none-any.whl">paramiko-5.0.0-py3-none-any.whl</a>                                                                     08-Jul-2026 08:35  208.9 kB
<a href="paramiko-5.0.0-py3-none-any.whl.metadata">paramiko-5.0.0-py3-none-any.whl.metadata</a>                                                            08-Jul-2026 08:35
<a href="paramiko-5.0.0.tar.gz">paramiko-5.0.0.tar.gz</a>                                                                               08-Jul-2026 08:35  1.5 MB
</pre><hr></body>

Same url on the proxies returns nothing:

$ curl -s https://fmproxy.internal/pulp/content/xxx/Library/custom/python/pypi/
$

I also recall when running Foreman 3.17/Katello 4.19 I could list the content from Foreman by adding “simple” to the path, ex:
$ curl -s https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/simple/

This however does no longer work either.

My repo settings is very basic:

Basic Information
Name: pypi
Label: pypi
Type: python

Sync Settings
Upstream URL: https://pypi.org
Excludes:
Includes: paramiko
Keep latest packages: 2
Mirroring Policy: Content Only

And as I said above, adding anything other then paramiko to the include list just breaks with the “No declared artifact” error.

I seen the “full sync” requirement in the documentation and it did not make a difference. I have also removed the old repo + ran “foreman-rake katello:delete_orphaned_content” and recreated it when running the latest version to avoid anything “old” messing things up.

I have the same problem.

Exception:

Katello::Errors::Pulp3Error: No declared artifact with relative path “click-6.1-py2.py3-none-any.whl.metadata” for content “<PythonPackageContent: click [6.1] (bdist_wheel)>”

System Information

Version 3.19.1

Katello 4.21.0

Oracle Linux Server release 9.8

I actually noticed my python content started to break already in the previous version Foreman 3.18.x/Katello 4.20.x. With this version I noticed I could sync all python content from Pypi but for some reason only ~40% of the content was properly synced to the proxies (no visable errors). When I seen this problem and the hint of Python was reworked in the new version I thought, maybe the new version will fix it…it made it worse :frowning:
So I would say the last version things actually worked properly is Foreman 3.17.x/Katello 4.19.x.

Can it be related to Pull-through caching does not create the metadata artifact for wheels · Issue #1087 · pulp/pulp_python ?

Are these new repos you’re creating on Katello 4.21 or older repos that are now failing. Also, are you seeing the same errors on sync on smart proxies for new repos? There is a slight difference between old and new repos with the upgrade. On the foreman server, we have an upgrade task that transforms the python repos into the new publication-less format pulp now supports.

For smart proxies, the suggested path is a complete sync which will resync everything. We should look at this as 2 separate issues, one on foreman server during pypi sync vs smart proxy sync.

I was able to sync a new python repo from pypi locally on that pulp-python version with some packages you listed above, with includes field being this for example:

aiobotocore==3.7.0
click==8.4.2

Does it fail in your setup for similar new repos?

So added these two lines:
aiobotocore==3.7.0
click==8.4.2
Sync was now successful!

I now readded my original 100+ python libraries that failed in the past and that sync was also successful now! So for some reason it started to work again. Have not changed anything on the Foreman server so pretty odd. Maybe the server ran some background jobs to clear things up?

So now if I check with (my product name is python with a repo name pypi):
curl -s https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/
I get back a page with a long list of expected .whl/whl.metadata/.tar.gz content so actually looks like that is working.

I see a “normal” quick sync (~1 minute completion time) was issued to my proxies after the pypi sync completed.
Checking with the same command on the proxy, it still returns nothing.
curl -s https://fmproxy.internal/pulp/content/xxx/Library/custom/python/pypi/

I however am not sure with this new version if that is expected to return something or not to check in the sync to my proxies is working.
I have now issued a “Full Sync” on one of the proxies and will report back when it has completed if that “did” anything.

After a full sync I get this error message now that I have not seen before that kind of make sense since looking on the Foreman server, the /simple/ path does not exist:

Error

Last sync failed: 404, message=‘Not Found’, url=‘https://foreman.internal/pypi/xxx/Library/custom/python/pypi/simple/

tests:
$ curl -s https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/simple/
404: Not Found
$ curl -s https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/| grep simple
$

Snippet from production.log for the full sync:

2026-07-17T06:37:44 [I|app|e4ab0ceb] Started POST "/katello/api/capsules/2-fmproxy-internal/content/sync" for 10.10.10.10 at 2026-07-17 06:37:44 +0000
2026-07-17T06:37:44 [I|app|e4ab0ceb] Processing by Katello::Api::V2::CapsuleContentController#sync as HTML
2026-07-17T06:37:44 [I|app|e4ab0ceb]   Parameters: {"id"=>"2-fmproxy-internal", "skip_metadata_check"=>true, "api_version"=>"v2", "capsule_content"=>{"id"=>"2-fmproxy-internal", "skip_metadata_check"=>true}}
2026-07-17T06:37:44 [I|bac|e4ab0ceb] Task {label: , execution_plan_id: 20cd8437-266f-4785-9f17-afa8dbc60036} state changed: pending
2026-07-17T06:37:44 [I|bac|e4ab0ceb] Task {label: Actions::Katello::CapsuleContent::Sync, id: a1bd2601-6f56-416b-900a-624196f525b7, execution_plan_id: 20cd8437-266f-4785-9f17-afa8dbc60036} state changed: planning
2026-07-17T06:37:55 [I|bac|e4ab0ceb] Task {label: Actions::Katello::CapsuleContent::Sync, id: a1bd2601-6f56-416b-900a-624196f525b7, execution_plan_id: 20cd8437-266f-4785-9f17-afa8dbc60036} state changed: planned
2026-07-17T06:37:55 [I|app|e4ab0ceb]   Rendered /usr/share/gems/gems/katello-4.21.0/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (Duration: 133.3ms | Allocations: 231036)
2026-07-17T06:37:55 [I|app|e4ab0ceb]   Rendered layout /usr/share/gems/gems/katello-4.21.0/app/views/katello/api/v2/layouts/resource.json.erb (Duration: 133.6ms | Allocations: 231132)
2026-07-17T06:37:55 [I|app|e4ab0ceb] Completed 202 Accepted in 11178ms (Views: 132.9ms | ActiveRecord: 759.5ms | Allocations: 4317215)
2026-07-17T06:37:57 [I|bac|e4ab0ceb] Task {label: Actions::Katello::CapsuleContent::Sync, id: a1bd2601-6f56-416b-900a-624196f525b7, execution_plan_id: 20cd8437-266f-4785-9f17-afa8dbc60036} state changed: running
2026-07-17T06:47:08 [E|bac|e4ab0ceb] 404, message='Not Found', url='https://foreman.internal/pypi/xxx/Library/custom/python/pypi/simple/' (Katello::Errors::Pulp3Error)
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:115:in `block in check_for_errors'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:113:in `each'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:113:in `check_for_errors'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:169:in `poll_external_task'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/polling.rb:22:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/cancellable.rb:15:in `run'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:26:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:33:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 e4ab0ceb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 e4ab0ceb | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/execution_wrapper.rb:92:in `wrap'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/progress.rb:15:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:33:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:48:in `restore_current_user'
 e4ab0ceb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/world.rb:31:in `execute'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:588:in `catch'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:588:in `block in execute_run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `block in with_error_handling'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `catch'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `with_error_handling'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:583:in `execute_run'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:304:in `execute'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/director.rb:95:in `execute'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors.rb:18:in `run_user_code'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 e4ab0ceb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 e4ab0ceb | [ sidekiq ]
 e4ab0ceb | [ concurrent-ruby ]
2026-07-17T07:13:51 [I|bac|e4ab0ceb] Task {label: Actions::Katello::CapsuleContent::Sync, id: a1bd2601-6f56-416b-900a-624196f525b7, execution_plan_id: 20cd8437-266f-4785-9f17-afa8dbc60036} state changed: stopped  result: warning
2026-07-17T07:13:51 [I|app|e4ab0ceb] ForemanWebhooks::EventSubscriber: actions.katello.capsule_content.sync_failed.event.foreman event received

Is this an upgraded smart proxy? There’s some change in behavior of smart proxy syncs. We generally expect a complete sync to fix all the old python repos to point to the correct distribution URL..Are these repos in Content views?

So just to recap, after the upgrade to Foreman 3.19.1/Katello 4.21 (both Foreman server and proxies) I removed all python repositories and + ran “foreman-rake katello:delete_orphaned_content” + performed a full sync.
I then readded the repos + full sync again. Not sure there is much more I can do?

What I find mostly “disturbing” is the fact that the “https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/simple/” path does not exist on the Foreman server and that seems to be the path the sync is trying to find but fails. Should the “/simple/” directory no longer be preset on a python repo? If it should, then the issue must be on the Foreman server itself and that is the reason for the failed proxy sync’s:

They way our users consume the python content with pip is by setting:

pip config --user set global.index https://fmproxy.internal/pulp/content/xxx/Library/custom/python/pypi
pip config --user set global.index-url https://fmproxy.internal/pulp/content/xxx/Library/custom/python/pypi/simple
pip config --user set global.trusted-host fmproxy.internal

So I guess pip kinda assume the index-url is the path containing the “/simple” path. So I would be a little surprised if this is not the same even in the new Foreman version.

This is one of the changes that are part of the release. The /pulp/content/ path for python repos doesn’t exist anymore. The correct path now would be a pypi endpoint..Something like foreman_url/pypi/my_repo..

The smart proxies when they run a complete sync should update their remote urls to point to new pypi endpoint. All new python repos in upgraded environment should be on the pypi endpoint.

So as a test I created a new product and a new python repo.
product name: python_test
repo name: pypi_2
Includes: abxpkg

sync worked fine and of course no content synced to my proxies.

I see “Published At” sets to “https://foreman.internal/pulp/content/xxx/Library/custom/python_test/pypi_2/
and if I go there, this is where I also see the content.

That is unexpected..It’s behaving like pre-upgrade python even though you’re on the right version 4.21. Can you check if you have a /pypi block in /etc/httpd/conf.d/05-foreman.conf apache config?

And if you can reach the same content at https://foreman.internal/pypi/xxx/Library/custom/python_test/pypi_2/

I see in /etc/httpd/conf.d/05-foreman.conf this pypi section:

  <Location "/pypi">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    RequestHeader set X-FORWARDED-PROTO expr=%{REQUEST_SCHEME}
    ProxyPass unix:///run/pulpcore-api.sock|http://pulpcore-api/pypi timeout=600
    ProxyPassReverse unix:///run/pulpcore-api.sock|http://pulpcore-api/pypi
  </Location>

On the that url I get:

curl -s https://foreman.internal/pypi/xxx/Library/custom/python_test/pypi_2/
{"projects":1,"releases":2,"files":4}

Trying the same command on my proxy returns the same “{“projects”:1,“releases”:2,“files”:4}”

Adding “simple/” to the url get me:

curl -s https://fmproxy.internal/pypi/xxx/Library/custom/python_test/pypi_2/simple/
<!DOCTYPE html>
<html>
  <head>
    <title>Simple Index</title>
    <meta name="pypi:repository-version" content="1.1">
  </head>
  <body>

      <a href="abxpkg/">abxpkg</a><br/>

  </body>

So that actually looks like the index url is working there here.

Trying the same set of commands on my production python repo:

curl -s https://foreman.internal/pypi/xxx/Library/custom/python/pypi/
{"projects":96,"releases":189,"files":1907}
curl -s https://foreman.internal/pypi/xxx/Library/custom/python/pypi/simple/
{"detail":"No PythonDistribution found for base_path xxx/Library/custom/python"}

Trying the proxy:

curl -s https://fmproxy.internal/pypi/xxx/Library/custom/python/pypi/
{"projects":0,"releases":0,"files":0}
curl -s https://fmproxy.internal/pypi/xxx/Library/custom/python/pypi/simple/
{"detail":"No PythonDistribution found for base_path xxx/Library/custom/python"}

So the new product/repo looks more healthy for sure. I guess I can again delete the python product and pypi repo and recreate them but I have this feeling there is perhaps something old remaining that gets me back to this broken state if I try and reuse the same product and repo name as I used before.

The 4.21 upgrade task should have handled the older python repos.The upgrade task to migrate old python repos can also be run as:
foreman-rake katello:upgrades:4.21:cleanup_python_publications

So more testing and no luck.
Removed all python repos

verified with:
foreman-rake katello:upgrades:4.21:cleanup_python_publications
No python repositories found. Skipping upgrade task.

Ran foreman-rake katello:delete_orphaned_content and waited for all “Remove orphans” to finish.

Recreated my repo:
Product: python
Repo: pypi
Upstream URL: https://pypi.org
Includes: abxpkg
Keep latest packages: 2

Foreman repo sync is successful but proxy sync fails with:

Last sync failed: 404, message=‘Not Found’, url=‘https://foreman.internal/pypi/xxx/Library/custom/python/pypi/simple/

$ curl -s https://foreman.internal/pypi/xxx/Library/custom/python/pypi/
{“projects”:1,“releases”:2,“files”:4}

$ curl -s https://foreman.internal/pypi/xxx/Library/custom/python/pypi/simple/
{“detail”:“No PythonDistribution found for base_path xxx/Library/custom/python”}

Tried the upgrade command:

foreman-rake katello:upgrades:4.21:cleanup_python_publications
Migrating Python distributions from publications to repository_versions...
Repairing Python package metadata...
Queued metadata repair for: pypi
Metadata repair completed for pypi
Deleting Python publications from Pulp database...
No Python publications found in Pulp database
Clearing publication_href references in Katello database...
No repositories with publication_href found

After that I still get:
curl -s https://foreman.internal/pypi/xxx/Library/custom/python/pypi/simple/
{“detail”:“No PythonDistribution found for base_path xxx/Library/custom/python”}

Issued a full sync to my proxy but I suspect this will end again with an error since the “pypi/xxx/Library/custom/python/pypi/simple/” seems broken on the server.