Did some of suggested tests but did not have any affect.
I had backups of Foreman and 5 of my proxies with 3.71/4.9.2 so I rolled back the machines, synced all repos and then performed a simple test by just adding one package to one of our production repos and measure the speed the meta data is synced to my proxies all over the world.
se = sweden where the foreman server is located. latency ~3 ms
us = USA, latency ~111 ms
ca = Canada, latency ~177 ms
cl = Chile, latency ~276 ms
au = Australia, latency ~420 ms
With versions 3.8/4.10
Synchronize smart proxy 'seproxy'|stopped|success|December 03, 2023 at 10:25:31 PM 1 minute|
Synchronize smart proxy 'usproxy'|stopped|success|December 03, 2023 at 10:25:34 PM 5 minutes|
Synchronize smart proxy 'caproxy'|stopped|success|December 03, 2023 at 10:25:31 PM 6 minutes|
Synchronize smart proxy 'clproxy'|stopped|success|December 03, 2023 at 10:25:32 PM 9 minutes|
Synchronize smart proxy 'auproxy'|stopped|success|December 03, 2023 at 10:25:35 PM 13 minutes|
With versions 3.71/4.9.2
Synchronize smart proxy 'seproxy' stopped success December 09, 2023 at 09:51:08 PM 5 seconds
Synchronize smart proxy 'usproxy' stopped success December 09, 2023 at 09:51:08 PM 17 seconds
Synchronize smart proxy 'caproxy' stopped success December 09, 2023 at 09:51:06 PM 25 seconds
Synchronize smart proxy 'clproxy' stopped success December 09, 2023 at 09:51:07 PM 36 seconds
Synchronize smart proxy 'auproxy' stopped success December 09, 2023 at 09:51:09 PM 50 seconds
As you can see the performance for 3.8/4.10 is horrific. Each time a package is released in our production repos, the sync is crazy slow and stacks up when several packages + syncs runs quickly after each other. Sadly I did not have full backup of all proxies so I guess to be able to go back to 3.71/4.9.2 on all proxies I need to redeploy 50% of my proxies unless there is a good way to downgrade a proxy…
Installed packages on the 3.71/4.9.2 Forman server.
Installed Packages
ansible-collection-theforeman-foreman-3.13.0-1.el8.noarch
ansiblerole-foreman_scap_client-0.2.0-2.el8.noarch
candlepin-4.3.1-1.el8.noarch
candlepin-selinux-4.3.1-1.el8.noarch
foreman-3.7.1-1.el8.noarch
foreman-cli-3.7.1-1.el8.noarch
foreman-debug-3.7.1-1.el8.noarch
foreman-dynflow-sidekiq-3.7.1-1.el8.noarch
foreman-installer-3.7.1-1.el8.noarch
foreman-installer-katello-3.7.1-1.el8.noarch
foreman-obsolete-packages-1.5-1.el8.noarch
foreman-postgresql-3.7.1-1.el8.noarch
foreman-proxy-3.7.1-1.el8.noarch
foreman-release-3.7.1-1.el8.noarch
foreman-selinux-3.7.1-1.el8.noarch
foreman-service-3.7.1-1.el8.noarch
katello-4.9.2-1.el8.noarch
katello-certs-tools-2.9.0-2.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.9.2-1.el8.noarch
katello-debug-4.9.2-1.el8.noarch
katello-repos-4.9.2-1.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
pulpcore-selinux-1.3.2-1.el8.x86_64
python39-pulp-ansible-0.16.0-1.el8.noarch
python39-pulp-certguard-1.5.6-1.el8.noarch
python39-pulp-cli-0.14.0-4.el8.noarch
python39-pulp-container-2.14.7-1.el8.noarch
python39-pulp-deb-2.20.4-1.el8.noarch
python39-pulp-file-1.12.0-1.el8.noarch
python39-pulp-python-3.8.0-1.el8.noarch
python39-pulp-rpm-3.19.9-1.el8.noarch
python39-pulpcore-3.22.15-1.el8.noarch
qpid-proton-c-0.37.0-1.el8.x86_64
rubygem-foreman-tasks-8.1.1-1.fm3_7.el8.noarch
rubygem-foreman_ansible-12.0.6-1.fm3_7.el8.noarch
rubygem-foreman_maintain-1.3.5-1.el8.noarch
rubygem-foreman_openscap-7.0.0-1.fm3_7.el8.noarch
rubygem-foreman_remote_execution-10.1.0-1.fm3_7.el8.noarch
rubygem-hammer_cli-3.7.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.7.0-1.el8.noarch
rubygem-hammer_cli_foreman_ansible-0.5.0-1.fm3_7.el8.noarch
rubygem-hammer_cli_foreman_openscap-0.1.13-2.fm3_5.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.3-1.fm3_7.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.19-1.fm3_7.el8.noarch
rubygem-hammer_cli_katello-1.9.1-1.el8.noarch
rubygem-katello-4.9.2-1.el8.noarch
rubygem-pulp_ansible_client-0.16.1-1.el8.noarch
rubygem-pulp_certguard_client-1.6.5-1.el8.noarch
rubygem-pulp_container_client-2.14.5-1.el8.noarch
rubygem-pulp_deb_client-2.20.2-1.el8.noarch
rubygem-pulp_file_client-1.12.0-1.el8.noarch
rubygem-pulp_ostree_client-2.0.0-1.el8.noarch
rubygem-pulp_python_client-3.8.0-1.el8.noarch
rubygem-pulp_rpm_client-3.19.6-1.el8.noarch
rubygem-pulpcore_client-3.22.4-1.el8.noarch
rubygem-qpid_proton-0.37.0-1.el8.x86_64
rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch