Import of most python content from Pypi is failing in Foreman 3.19.1/Katello 4.21

Just upgraded to Foreman 3.19.1/Katello 4.21 and noticed that my Python product that contain a repository syncing python content from Pypi is not working anymore. tired adding some different libraries to the Includes list and most if them fail with a similar error. Ex.

aiobotocore:
No declared artifact with relative path “aiobotocore-3.7.0-py3-none-any.whl.metadata” for content “<PythonPackageContent: aiobotocore [3.7.0] (bdist_wheel)>”

click:
No declared artifact with relative path “click-8.4.1-py3-none-any.whl.metadata” for content “<PythonPackageContent: click [8.4.1] (bdist_wheel)>”

ansible_pylibssh:
No declared artifact with relative path “ansible_pylibssh-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata” for content “<PythonPackageContent: ansible-pylibssh [1.3.0] (bdist_wheel)>”

One library that I did manage to sync is paramiko but most other seems to fail.

Some logs:

from journalctl:

journalctl -u 'pulpcore*' -S "2026-07-08 07:45:00" -o cat --no-pager
`
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:14 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 7696 "-" "rest-client/2.1.0 (linux x86_64) ruby/3.0.7p220"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Starting task id: 019f40b0-2a17-7c66-bce4-0d6948dcf007 in domain: default, task_type: pulpcore.app.tasks.base.ageneral_update, immediate: True, deferred: True
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Task completed 019f40b0-2a17-7c66-bce4-0d6948dcf007 in domain: default, task_type: pulpcore.app.tasks.base.ageneral_update, immediate: True, deferred: True, execution_time: 11786 μs
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:14 +0000] "PATCH /pulp/api/v3/remotes/python/python/019f40a6-8afa-7288-b28e-6e92fa417e3d/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:14 +0000] "GET /pulp/api/v3/tasks/019f40b0-2a17-7c66-bce4-0d6948dcf007/ HTTP/1.1" 200 1756 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:15 +0000] "POST /pulp/api/v3/repositories/python/python/019f40a6-8c3b-7d5d-9779-14ccb6e48aa8/sync/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Starting task id: 019f40b0-2aed-79f0-bd7e-9caf3fe6f34f in domain: default, task_type: pulp_python.app.tasks.sync.sync, immediate: False, deferred: True
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized project plugin allowlist_project, filtering ['ansible-pylibssh']
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized release plugin allowlist_release, filtering [<Requirement('ansible-pylibssh')>]
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized release plugin blocklist_release, filtering []
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized prerelease plugin with [re.compile('.+rc\\d+$'), re.compile('.+a(lpha)?\\d+$'), re.compile('.+b(eta)?\\d+$'), re.compile('.+dev\\d+$')]
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch:INFO: Initialized prerelease plugin prerelease_release, filtering all packages
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.mirror:INFO: Syncing with https://pypi.org.
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.mirror:INFO: No metadata filters are enabled. Skipping metadata filtering
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.mirror:INFO: No release file filters are enabled. Skipping release file filtering
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:15 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1233 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: bandersnatch.package:INFO: Fetching metadata for package: ansible-pylibssh (serial 0)
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:15 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1233 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:16 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1235 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:16 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1235 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:17 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1237 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:18 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 1237 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO: Task[pulp_python.app.tasks.sync.sync] 019f40b0-2aed-79f0-bd7e-9caf3fe6f34f failed (ValueError: No declared artifact with relative path "ansible_pylibssh-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata" for content "<PythonPackageContent: ansible-pylibssh [1.3.0] (bdist_wheel)>") in domain: default
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]: pulpcore.tasking.tasks:INFO:   File "/usr/lib/python3.12/site-packages/pulpcore/tasking/tasks.py", line 73, in _execute_task
    result = task_function()
             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pulp_python/app/tasks/sync.py", line 60, in sync
    DeclarativeVersion(first_stage, repository, mirror).create()
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/declarative_version.py", line 163, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/api.py", line 217, in create_pipeline
    await asyncio.gather(*futures)
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/api.py", line 40, in __call__
    await self.run()
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 318, in run
    await self._handle_remote_artifacts(batch)
  File "/usr/lib/python3.12/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 418, in _handle_remote_artifacts
    raise ValueError(
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "GET /pulp/api/v3/tasks/019f40b0-2aed-79f0-bd7e-9caf3fe6f34f/ HTTP/1.1" 200 2648 "-" "OpenAPI-Generator/3.105.5/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "GET /pulp/api/v3/distributions/python/pypi/?base_path=ssc%2FLibrary%2Fcustom%2Fpython%2Fpypi HTTP/1.1" 200 759 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "GET /pulp/api/v3/distributions/python/pypi/019f40a6-8f01-7e98-b61a-12155ea15a7f/ HTTP/1.1" 200 707 "-" "OpenAPI-Generator/3.27.2/ruby"
pulp [7d229deb-e2a5-466d-ac7a-463b3ad11a9b]:  - - [08/Jul/2026:07:45:19 +0000] "PATCH /pulp/api/v3/distributions/python/pypi/019f40a6-8f01-7e98-b61a-12155ea15a7f/ HTTP/1.1" 200 707 "-" "OpenAPI-Generator/3.27.2/ruby"

production log:

grep 7d229deb production.log
2026-07-08T07:45:14 [I|app|7d229deb] Started POST "/katello/api/v2/repositories/1303/sync?organization_id=1" for 10.107.17.18 at 2026-07-08 07:45:14 +0000
2026-07-08T07:45:14 [I|app|7d229deb] Processing by Katello::Api::V2::RepositoriesController#sync as HTML
2026-07-08T07:45:14 [I|app|7d229deb]   Parameters: {"id"=>"1303", "organization_id"=>"1", "api_version"=>"v2", "repository"=>{"id"=>"1303"}}
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: , execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: pending
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: planning
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: planned
2026-07-08T07:45:14 [I|app|7d229deb]   Rendered /usr/share/gems/gems/katello-4.21.0/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 18.1ms | Allocations: 14313)
2026-07-08T07:45:14 [I|app|7d229deb]   Rendered layout /usr/share/gems/gems/katello-4.21.0/app/views/katello/api/v2/layouts/resource.json.erb (Duration: 18.3ms | Allocations: 14406)
2026-07-08T07:45:14 [I|app|7d229deb] Completed 202 Accepted in 280ms (Views: 17.8ms | ActiveRecord: 26.2ms | Allocations: 98026)
2026-07-08T07:45:14 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: running
2026-07-08T07:45:19 [E|bac|7d229deb] No declared artifact with relative path "ansible_pylibssh-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata" for content "<PythonPackageContent: ansible-pylibssh [1.3.0] (bdist_wheel)>" (Katello::Errors::Pulp3Error)
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:115:in `block in check_for_errors'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:113:in `each'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:113:in `check_for_errors'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:169:in `poll_external_task'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/polling.rb:22:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/cancellable.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:33:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 7d229deb | /usr/share/gems/gems/katello-4.21.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 7d229deb | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/execution_wrapper.rb:92:in `wrap'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action/progress.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:33:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:48:in `restore_current_user'
 7d229deb | /usr/share/gems/gems/foreman-tasks-12.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/world.rb:31:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:588:in `catch'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:588:in `block in execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `block in with_error_handling'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `catch'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `with_error_handling'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:583:in `execute_run'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:304:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/director.rb:95:in `execute'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors.rb:18:in `run_user_code'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 7d229deb | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 7d229deb | [ sidekiq ]
 7d229deb | [ concurrent-ruby ]
2026-07-08T07:45:19 [I|bac|7d229deb] Task {label: Actions::Katello::Repository::Sync, id: 7a38df45-86c8-4c4c-a3c4-08430fd78b33, execution_plan_id: 35ea189b-3e4b-4e7a-b104-2d861164d197} state changed: stopped  result: warning
2026-07-08T07:45:19 [I|app|7d229deb] ForemanWebhooks::EventSubscriber: actions.katello.repository.sync_failed.event.foreman event received

versions:

# rpm -qa | sort | egrep 'katello|foreman|pulpcore|pulp-python'
ansible-collection-theforeman-foreman-5.11.0-1.el9.noarch
ansiblerole-foreman_scap_client-0.5.0-1.el9.noarch
foreman-3.19.1-1.el9.noarch
foreman-cli-3.19.1-1.el9.noarch
foreman-dynflow-sidekiq-3.19.1-1.el9.noarch
foreman-installer-3.19.1-1.el9.noarch
foreman-installer-katello-3.19.1-1.el9.noarch
foreman-obsolete-packages-1.11-2.el9.noarch
foreman-postgresql-3.19.1-1.el9.noarch
foreman-proxy-3.19.1-1.el9.noarch
foreman-redis-3.19.1-1.el9.noarch
foreman-release-3.19.1-1.el9.noarch
foreman-selinux-3.19.1-1.el9.noarch
foreman-service-3.19.1-1.el9.noarch
katello-4.21.0-1.el9.noarch
katello-certs-tools-2.10.0-1.el9.noarch
katello-client-bootstrap-1.7.9-2.el9.noarch
katello-common-4.21.0-1.el9.noarch
katello-repos-4.21.0-1.el9.noarch
katello-selinux-5.2.0-1.el9.noarch
pulpcore-obsolete-packages-1.3.0-1.el9.noarch
pulpcore-selinux-2.2.0-1.el9.x86_64
python3.12-pulpcore-3.105.6-1.el9.noarch
python3.12-pulp-python-3.27.2-1.el9.noarch
rubygem-foreman_ansible-17.0.5-1.fm3_19.el9.noarch
rubygem-foreman_maintain-1.15.0-1.el9.noarch
rubygem-foreman_openscap-12.1.0-1.fm3_19.el9.noarch
rubygem-foreman_remote_execution-16.6.4-1.fm3_19.el9.noarch
rubygem-foreman-tasks-12.1.1-1.fm3_19.el9.noarch
rubygem-foreman_webhooks-5.0.3-1.fm3_19.el9.noarch
rubygem-hammer_cli_foreman-3.19.0-1.el9.noarch
rubygem-hammer_cli_foreman_ansible-0.8.2-1.fm3_18.el9.noarch
rubygem-hammer_cli_foreman_openscap-0.2.3-1.fm3_17.el9.noarch
rubygem-hammer_cli_foreman_remote_execution-0.4.2-1.fm3_19.el9.noarch
rubygem-hammer_cli_foreman_tasks-0.0.24-1.fm3_17.el9.noarch
rubygem-hammer_cli_katello-1.21.0-1.el9.noarch
rubygem-katello-4.21.0-1.el9.noarch
rubygem-pulpcore_client-3.105.5-1.el9.noarch

Not sure if this is the caused, but did you run a full sync for the python content as advised in Katello 4.21.0 is now available

Can also add that if I only add paramiko to the include list and the sync works, it is only available on the Foreman server and not on the proxies (I see proxy sync tasks run and complete OK after the sync) .
ex:

$ curl -s https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/

<html>
<head><title>Index of /pulp/content/xxx/Library/custom/python/pypi/</title></head>
<body bgcolor="white">
<h1>Index of /pulp/content/xxx/Library/custom/python/pypi/</h1>
<hr><pre><a href="../">../</a>
<a href="paramiko-4.0.0-py3-none-any.whl">paramiko-4.0.0-py3-none-any.whl</a>                                                                     08-Jul-2026 08:35  223.9 kB
<a href="paramiko-4.0.0-py3-none-any.whl.metadata">paramiko-4.0.0-py3-none-any.whl.metadata</a>                                                            08-Jul-2026 08:35
<a href="paramiko-4.0.0.tar.gz">paramiko-4.0.0.tar.gz</a>                                                                               08-Jul-2026 08:35  1.6 MB
<a href="paramiko-5.0.0-py3-none-any.whl">paramiko-5.0.0-py3-none-any.whl</a>                                                                     08-Jul-2026 08:35  208.9 kB
<a href="paramiko-5.0.0-py3-none-any.whl.metadata">paramiko-5.0.0-py3-none-any.whl.metadata</a>                                                            08-Jul-2026 08:35
<a href="paramiko-5.0.0.tar.gz">paramiko-5.0.0.tar.gz</a>                                                                               08-Jul-2026 08:35  1.5 MB
</pre><hr></body>

Same url on the proxies returns nothing:

$ curl -s https://fmproxy.internal/pulp/content/xxx/Library/custom/python/pypi/
$

I also recall when running Foreman 3.17/Katello 4.19 I could list the content from Foreman by adding “simple” to the path, ex:
$ curl -s https://foreman.internal/pulp/content/xxx/Library/custom/python/pypi/simple/

This however does no longer work either.

My repo settings is very basic:

Basic Information
Name: pypi
Label: pypi
Type: python

Sync Settings
Upstream URL: https://pypi.org
Excludes:
Includes: paramiko
Keep latest packages: 2
Mirroring Policy: Content Only

And as I said above, adding anything other then paramiko to the include list just breaks with the “No declared artifact” error.

I seen the “full sync” requirement in the documentation and it did not make a difference. I have also removed the old repo + ran “foreman-rake katello:delete_orphaned_content” and recreated it when running the latest version to avoid anything “old” messing things up.

I have the same problem.

Exception:

Katello::Errors::Pulp3Error: No declared artifact with relative path “click-6.1-py2.py3-none-any.whl.metadata” for content “<PythonPackageContent: click [6.1] (bdist_wheel)>”

System Information

Version 3.19.1

Katello 4.21.0

Oracle Linux Server release 9.8

I actually noticed my python content started to break already in the previous version Foreman 3.18.x/Katello 4.20.x. With this version I noticed I could sync all python content from Pypi but for some reason only ~40% of the content was properly synced to the proxies (no visable errors). When I seen this problem and the hint of Python was reworked in the new version I thought, maybe the new version will fix it…it made it worse :frowning:
So I would say the last version things actually worked properly is Foreman 3.17.x/Katello 4.19.x.