After thinking about this for a long time, why would it not work with 4k servers when this is so small number compared to Foreman capacity it came to me that in 2.0 I applied a patch for a BUG where packages were not showing. Could not calculate errata status /no packake info/ Foreman 2.0/Katello 3.15
I presumed, with that patch and upgrade to new version something has to be broken in DB.
So I decided to cut my losses, unsubscribe all the machines, create new fresh 2.1 install and subscribe them all back.
So far I’ve subscribed 7k machines and absolutely no issues at all. I even stress tested it with running simultaneous subscription of 100 machines at a time, with 6k already registered. The highest average response time I could get it was 700 ms. When no subscribing or jobs are ran, average response is under 50 ms. Fresh install works like a charm.
We can close this. However I seem to have found a bug now regarding pulp3. Pulp 3 can't work with yum repo requiring auth/Foreman 2.1.2