Cannot install packages from proxy "KeyError: 'md5'"

Problem:
cannot download rpm from proxy

 Transferred a partial file for https://katello-proxy-01.example.com/pulp/content/provalida/PROD/CCV_PV_AlmaLinux_8/custom/AlmaLinux_8/AppStream_x86_64_os/Packages/l/libfontenc-1.1.3-8.el8.x86_64.rpm [transfer closed with outstanding read data remaining]

Foreman and Proxy versions:
3.13 / 4.15
Foreman and Proxy plugin versions:

Distribution and version:
Almalinux 9.5

Other relevant data:
this issues just occured when i’m trying to install java-17 for example

When i’m triyin to install java-17 from another Host, which is not using this Proxy, everything is fine
I can’t even download it via Browser or curl
But on Katello Server and Proxy-2 the issue cannot be reproduced, installation of java-17 is working

affected Packages at the moment:
libfontenc-1.1.3-8.el8.x86_64.rpm
ttmkfdir-3.0.9-54.el8.x86_64.rpm

I tried the following:
rerun foreman-installer
Verify Content Checksum on affected Repo
Republich Repository Metadata on affected Repo
Proxy FULL SYNC

Issue still not resolved, katello Server and another Proxy don’t have this issue

StackTrace:

Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     resp = await handler(request)
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:            ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 352, in cached_function
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     response = await self.make_entry(
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:                ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 395, in make_entry
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     response = await handler(*args, **kwargs)
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 284, in stream_content
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     return await self._match_and_stream(path, request)
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 687, in _match_and_stream
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     return await self._stream_content_artifact(
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 847, in _stream_content_artifact
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     response = await self._stream_remote_artifact(request, response, remote_artifact)
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 1150, in _stream_remote_artifact
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     download_result = await downloader.run()
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:                       ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/download/http.py", line 269, in run
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     return await download_wrapper()
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:            ^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/backoff/_async.py", line 151, in retry
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     ret = await target(*args, **kwargs)
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/download/http.py", line 254, in download_wrapper
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     return await self._run(extra_data=extra_data)
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulp_rpm/app/downloaders.py", line 118, in _run
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     to_return = await self._handle_response(response)
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/download/http.py", line 205, in _handle_response
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     await self.finalize()
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 1141, in finalize
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     await original_finalize()
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/download/base.py", line 171, in finalize
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     self.validate_digests()
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:   File "/usr/lib/python3.11/site-packages/pulpcore/download/base.py", line 221, in validate_digests
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:     actual_digest = self._digests[algorithm].hexdigest()
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]:                     ~~~~~~~~~~~~~^^^^^^^^^^^
Mar 06 12:01:13 katello-proxy-01.example.com pulpcore-content[417265]: KeyError: 'md5'

Activity