Best way to deploy foreman for support of ~ 6000 instances

Hello guys! We have approximately 6000 instances, managed by puppet, and we are thinking of moving all this structure (with our puppet classes - most of them are custom) to foreman. We have a multi-regional structure, and I think we are just going to use one proxy host for a region, which will be something like 600-800 instances. I have some concerns - how big proxy instance do we need? Will foreman (and foreman UI) handle such amount of hosts/connections? I think we are going to synchronize instances in the 20-minute interval. I just tried foreman in a test environment, so probably you can point some caveats or something which I should be aware of.

I am at 18,000 hosts currently. I know others are much higher.

My limiting factor/scaling headache is almost always puppet, not foreman

Java Heap. Heap
Heap.

More environments mean more heap as each env “duplicates” all your code, or more HEAP
More modules means more code overall, again more HEAP
More jruby threads means more heap as each thread has its own in memory process.

Scaling is about jruby threads. More is better concurrency, and more means more HEAP.

We currently do 18x puppetmasters with 18x jruby threads each but are looking to add 25% more. We have 35 environments with about 1100 classes per env. This translates to a HEAP of around 50gb to run those 18 jruby processs on each puppet node.

My foreman side to handle the ENC lookups for all this? Like 12 servers each with 8 to 12 httpd processes for foreman. And that’s just so I’m super redundant in terms of resiliency.

1 Like