Proxy Sync error, no valid Release file

Problem:
When I try to sync any repos over to the my proxy, I get an error saying no valid Release file.
I thought this might be an issue with the GPG keys, so I’ve removed all my repos and started fresh with just one deb repo and did not enable the GPG key.
The data seems to download from the Ubuntu archive when I sync from there, but the issue seems to be when I try to sync the data to the proxy.
I encountered this issue while on version 3.6 I have upgraded to 3.11.0.
I followed the instructions on upgrading and upgraded sequentially 3.6 to 3.7 to 3.8 to 3.9 to 3.10 to 3.11


Could not find a Release file at 'https://<my foreman server address>/deb/org_mywork_edu/Library/cv_UBUN2204_SOE/custom/Ubuntu_22_04/jammy-backports/dists/jammy-backports', try checking the 'url' and 'distributions' option on your remote Could not lookup a publication_href for repo 187

When I check the url, the “Release” file appears to to be there using the “Published At” url.
But not if using the URL with the content view which is the one being used when synced to the proxy.
From the proxy server I can grab the file Release file using curl using the “Published At” ruling out network issue.

curl -Is https://XXXXXXX/pulp/content/org_XXXXXX/Library/custom/Ubuntu_22_04/jammy-backports/dists/jammy-backports/Release | head -n20
HTTP/2 200 
date: Thu, 04 Jul 2024 06:31:06 GMT
server: Python/3.11 aiohttp/3.9.2
x-pulp-cache: HIT
content-type: application/octet-stream
etag: "17deecdf589798c6-2474"
last-modified: Thu, 04 Jul 2024 05:55:52 GMT
content-length: 9332
accept-ranges: bytes
via: 2.0 XXXXX.edu

some logs from /var/log/meesages .

Jul  4 15:56:49 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:49 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 6478 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.8p225"
Jul  4 15:56:50 esol-for01-pn pulpcore-api[31362]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Starting task 01907c51-2157-7b49-bcb4-7637127da2b6
Jul  4 15:56:50 esol-for01-pn pulpcore-api[31362]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Task completed 01907c51-2157-7b49-bcb4-7637127da2b6
Jul  4 15:56:50 esol-for01-pn pulpcore-api[31362]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:50 +0000] "PATCH /pulp/api/v3/remotes/deb/apt/01907c40-e640-704b-863e-5d41ff657a5f/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.2.0/ruby"
Jul  4 15:56:50 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:50 +0000] "GET /pulp/api/v3/tasks/01907c51-2157-7b49-bcb4-7637127da2b6/ HTTP/1.1" 200 750 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:50 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:50 +0000] "POST /pulp/api/v3/repositories/deb/apt/01907c40-e78e-77b8-b238-2ca6617ac9a0/sync/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.2.0/ruby"
Jul  4 15:56:50 esol-for01-pn pulpcore-worker-2[39293]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Starting task 01907c51-2258-7aee-9b29-346c75fba7bd
Jul  4 15:56:50 esol-for01-pn pulpcore-worker-2[39293]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulp_deb.app.tasks.synchronizing:INFO: Downloading Release file for distribution: "jammy-backports"
Jul  4 15:56:50 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:50 +0000] "GET /pulp/api/v3/tasks/01907c51-2258-7aee-9b29-346c75fba7bd/ HTTP/1.1" 200 866 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:51 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:51 +0000] "GET /pulp/api/v3/tasks/01907c51-2258-7aee-9b29-346c75fba7bd/ HTTP/1.1" 200 866 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:51 esol-for01-pn pulpcore-api[31362]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:51 +0000] "GET /pulp/api/v3/tasks/01907c51-2258-7aee-9b29-346c75fba7bd/ HTTP/1.1" 200 866 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:52 esol-for01-pn pulpcore-api[31362]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:52 +0000] "GET /pulp/api/v3/tasks/01907c51-2258-7aee-9b29-346c75fba7bd/ HTTP/1.1" 200 866 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:52 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:52 +0000] "GET /pulp/api/v3/tasks/01907c51-2258-7aee-9b29-346c75fba7bd/ HTTP/1.1" 200 866 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:53 esol-for01-pn pulpcore-api[31354]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:53 +0000] "GET /pulp/api/v3/tasks/01907c51-2258-7aee-9b29-346c75fba7bd/ HTTP/1.1" 200 866 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:54 esol-for01-pn pulpcore-worker-2[39293]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulp_deb.app.tasks.synchronizing:INFO: No change in ReleaseFiles detected. Skipping sync.
Jul  4 15:56:54 esol-for01-pn pulpcore-worker-2[39293]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Task completed 01907c51-2258-7aee-9b29-346c75fba7bd
Jul  4 15:56:54 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:54 +0000] "GET /pulp/api/v3/tasks/01907c51-2258-7aee-9b29-346c75fba7bd/ HTTP/1.1" 200 1049 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:54 esol-for01-pn pulpcore-api[31390]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:54 +0000] "GET /pulp/api/v3/publications/deb/apt/01907c50-1c54-7e24-97c2-bcf3a7666bbd/ HTTP/1.1" 200 430 "-" "OpenAPI-Generator/3.2.0/ruby"
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:55 +0000] "GET /pulp/api/v3/distributions/deb/apt/?base_path=org_mywork_edu%2FLibrary%2Fcustom%2FUbuntu_22_04%2Fjammy-backports HTTP/1.1" 200 623 "-" "OpenAPI-Generator/3.2.0/ruby"
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31354]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:55 +0000] "GET /pulp/api/v3/publications/deb/apt/01907c50-1c54-7e24-97c2-bcf3a7666bbd/ HTTP/1.1" 200 430 "-" "OpenAPI-Generator/3.2.0/ruby"
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Starting task 01907c51-3515-799a-8c05-a300d96013db
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Task completed 01907c51-3515-799a-8c05-a300d96013db
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:55 +0000] "PATCH /pulp/api/v3/distributions/deb/apt/01907c41-0372-701a-a422-6ba3d0b89b5f/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.2.0/ruby"
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31341]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:55 +0000] "GET /pulp/api/v3/tasks/01907c51-3515-799a-8c05-a300d96013db/ HTTP/1.1" 200 706 "-" "OpenAPI-Generator/3.49.6/ruby"
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:55 +0000] "GET /pulp/api/v3/publications/deb/apt/01907c50-1c54-7e24-97c2-bcf3a7666bbd/ HTTP/1.1" 200 430 "-" "OpenAPI-Generator/3.2.0/ruby"
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Starting task 01907c51-3689-7d05-863f-cd27124fed46
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]: pulpcore.tasking.tasks:INFO: Task completed 01907c51-3689-7d05-863f-cd27124fed46
Jul  4 15:56:55 esol-for01-pn pulpcore-api[31399]: pulp [cf52ce0c-cf6d-4bfb-afea-b149f9117508]:  - - [04/Jul/2024:05:56:55 +0000] "PATCH /pulp/api/v3/distributions/deb/apt/01907c41-0372-701a-a422-6ba3d0b89b5f/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.2.0/ruby"

Expected outcome:
Syncs to the proxy do not error.

Foreman and Proxy versions:
Both Foreman server and proxy server are at 3.11.0

Foreman and Proxy plugin versions:

Foreman server

 rpm -qa | grep foreman
foreman-release-3.11.0-1.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.21-1.fm3_11.el8.noarch
foreman-debug-3.11.0-1.el8.noarch
rubygem-foreman_remote_execution-13.1.0-1.fm3_11.el8.noarch
rubygem-foreman_ansible-14.0.0-2.fm3_11.el8.noarch
foreman-installer-3.11.0-1.el8.noarch
foreman-service-3.11.0-1.el8.noarch
foreman-selinux-3.11.0-1.el8.noarch
foreman-cli-3.11.0-1.el8.noarch
rubygem-foreman-tasks-9.1.1-1.fm3_11.el8.noarch
foreman-postgresql-3.11.0-1.el8.noarch
foreman-dynflow-sidekiq-3.11.0-1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman-3.11.0-1.el8.noarch
foreman-installer-katello-3.11.0-1.el8.noarch
foreman-proxy-3.11.0-1.el8.noarch
foreman-3.11.0-1.el8.noarch
foreman-obsolete-packages-1.9-1.el8.noarch
rubygem-foreman_maintain-1.6.9-1.el8.noarch
foreman-redis-3.11.0-1.el8.noarch
rpm -qa | grep  katello
rubygem-katello-4.13.0-1.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.13.0-1.el8.noarch
katello-certs-tools-2.10.0-1.el8.noarch
rubygem-hammer_cli_katello-1.13.0-0.2.pre.master.el8.noarch
katello-repos-4.13.0-1.el8.noarch
katello-debug-4.13.0-1.el8.noarch
foreman-installer-katello-3.11.0-1.el8.noarch
katello-4.13.0-1.el8.noarch
katello-ca-consumer-XXXX-1.0-3.noarch

Proxy Server:

 rpm -qa | grep foreman
foreman-release-3.11.0-1.el8.noarch
foreman-debug-3.11.0-1.el8.noarch
foreman-proxy-3.11.0-1.el8.noarch
foreman-installer-3.11.0-1.el8.noarch
foreman-installer-katello-3.11.0-1.el8.noarch
rpm -qa | grep katello
katello-certs-tools-2.10.0-1.el8.noarch
katello-ca-consumer-XXXXXXXX-1.0-5.noarch
katello-debug-4.13.0-1.el8.noarch
foreman-installer-katello-3.11.0-1.el8.noarch
katello-repos-4.13.0-1.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch

Distribution and version:
Red Hat Enterprise Linux release 8.10 (Ootpa)

Other relevant data:



Did you say that on the server, the Release file in question is present at the Library/custom/... location, but missing at the Library/cv_UBUN2204_SOE/custom/... location?

If so, could you see if there are any failed tasks relating to publishing or promoting the content view? Or perhaps some failed “Metadata generate” tasks?