Hey there !
I don’t really have a specific answer to your problems but I have some insights about a similar situation.
We have an “old” Foreman 1.11 with a second smart proxy maintaining the configuration of roughly 6000 servers. 12 different Linux flavors, 4 Windows versions. Split between bare metal and 3 different Hypervisors for virtualization.
The poor thing was in bad shape, on an OS that the company was no longer willing to spent time maintaining, and we were having performance issues. Long story short, we had to find a solution.
Migrate the old infrastructure or build a new one.
TLDR; We were not able to migrate with certainty and ease, and it would have been at a huge cost (both man hours and financially). We built a new infrastructure and left the “old” datas to die with the old platform while migrating all the hosts with Ansible playbooks.
So, the long story …
We have an old Foreman 1.11 and a secondary smart proxy, running outdated Puppet servers and agents packages. We have both version 3 and 4 of the puppet agent running on all the hosts. The database running all “that” is on MySQL.
The work was to migrate all “that” to an up to date OS with up to date packages for the OS and the softwares (Puppet/Foreman). Also, plan for upgrading all the hosts with the new up to date agent shortly after that (because version 3 and 4 agents could not talk to the new puppet servers).
So after having analyzed the work to do, which was migrating across 3 OS, 2 database technologies and several other little adjustments, we decided it was way too much work to migrate the old platform to a new one.
Instead we decided to build a new one, up to standards and with all the bells and whistles
We currently are in the process of migrating all the hosts on the old platform to the new one (Foreman 3.4/ Puppet 7), and only 40% of the total hosts are left to be migrated. You know how it is with “change management”
We have the remote execution feature up and running (on Linuxes) and are in the process of enabling the “build” feature.
We are are using custom tailored Ansible playbooks (nothing too fancy tho) to ensure that the migration of the hosts is proceeding in an orderly fashion. And we are migrating hosts at a pace of 200 a week ATM with ease. We are also taking some time to tweak the old Puppet code and bring it to the current standards ad well.
Everything is relying on Git repos and is versioned.
Hopes this will give some insights.