as you may have seen, there have been a few discussions around the memory usage of Foreman (and related parts of the stack):
While we think we have fixed the source for the particular increased memory usage in 2.5/3.0, we also found that our current tuning defaults for deployments aren’t as well defined as they could. This is mostly due to the fact that we have ways to do calculations of the form of “if you have 4 CPUs and 32GB of memory, you shouldn’t run more than 6 Puma workers” these calculations aren’t based on recent numbers and platform changes and might be suboptimal.
Let me quickly recap the current architecture:
Foreman
Apache HTTPd: Doing, well, HTTP. So SSL termination, forwarding requests to backends using mod_proxy, no mod_wsgi or mod_passenger anymore
Puma: The Ruby application server, running the main Ruby on Rails app.
PostgreSQL: The database.
Dynflow: The “Foreman jobs daemon”, doing various background tasks that the Rails app shouldn’t be blocked on.
Redis: Mostly used as a cache for Dynflow and Rails
Foreman Proxy
Foreman Proxy: Small Sinatra web app, running on WEBRick, responsible for integrating with external services like DNS, DHCP, Puppet etc.
Puppet Server
Puppet Server: If you’re using Puppet, you’ll also have a Puppet Server running on your machine
Katello
If you’re using Katello, you’ll also have the following additional services running on your system:
Pulpcore API: Django app (running in gunicorn) allows Foreman/Katello to manage Pulp repositories
Pulpcore Content: Django app (running in gunicorn) serves content (packages etc) to clients
Pulpcore Workers: Django app (running standalone) that does the “work” of downloading packages, generating repository metadata etc
Tomcat: Java application server, running the Candlepin app, responsible for all things subscriptions for Katello and Clients
Now, as you can see, depending on the deployed featureset, the number of services (and thus required resources) greatly differ. Additionally, y’all will have different use cases and use patterns, resulting in different resource usage, even with the same features enabled.
What we’re looking for are numbers to further improve our defaults. Mostly memory and CPU usage of the individual processes, plus a bit of details about your deployment.
Examples:
Foreman without Katello and Puppet on 4 CPU, 16GB RAM VM. 6 Puma workers at 1GB each, PostgreSQL 800MB, Redis 200MB, running flawlessly
Foreman with Katello and Puppet on 8 CPU, 32GB RAM. 12 Puma workers at 2GB each, PostgreSQL at 3GB, Redis at 1GB, Tomcat at 2GB and Puppet Server at 4GB, constantly OOMing
I have no access to a customer to give you all the information at the moment, but I have asked at least two which I was in contact in the last weeks.
For me a tuning preset which will give you the absolute minimum would be great, so I can create a local demo without blocking all resources for it. I just like to have my demos with me to be independent of internet and datacenter. Name this preset like demo or “minimal - no productive use” to make it clear, but it would be nice to have it (even after I order a new system with 32GB RAM ).
The added value of this might be a bit low, more in the domain of ‘good to know’:
Foreman + Katello, no Puppet:
Lab setting: 2 CPUs, 12 GB RAM. It runs 3 Puma’s and when I run free -m I see:
[ansible@foreman ~]$ free -m
total used free shared buff/cache available
Mem: 11800 5373 4863 247 1563 5901
Swap: 6063 0 6063
But it doesn’t serve more then 5 clients (my laptop can’t handle that )
We’re still waiting for our client to provision VMs, networks and firewall access, but the ‘real’ deployment of this lab is scaled at 4 CPUs and 24GB RAM. This should eventually server ~60 nodes, direct or through 4 smart proxies (2CPUs / 8 GB RAM)
Foreman + Katello + Puppet, running on 16 CPU Cores (currently @ ~35% CPU load on average during regular operation) and 64 GB RAM VM. 8 PUMA workers at ~800 MB each, PostgreSQL around 4GB of RAM, Redis at neglectable 150MB, Tomcat using ~4GB of RAM.
We have 3 smart-proxies solelly serving puppet and another 3 for content/packages. This setup currently serves just short of 3000 Hosts currently with a Puppet run interval of 1 hour.
We just updated this production environment to Foreman 2.5 this weekend from 2.0, so our sizing currently is still mainly based off the mod_passenger setup. Our sizing is also intended to withstand peaking requests, both for puppet runs and package updates, but we have not had any of those peaks since the update.
In our current day-to-day usage, this setup by now runs flawlessly, but we have observed random occurances of “slowness”, probably related to the way puma request queues work different from mod_passenger request queues. Before, when a user found Foreman to be unresponsive, this indicated requests piling up in general. Now they seem to just be randomly assigned to a worker that is currently handling more work than others while the system in general is completely fine.
One observation, I test Satellite on my ex-workstation with (I know I know I know) 12GB of RAM. It was quite decent workstation back then. Anyway:
This is Xeon W3550 with 4 cores and it worked fine. When we started shipping puppet server (recent versions) it sometimes had to swap, since I do not use Puppet I always turned it off. And it worked just fine.
Until recently when I realized things got much slower when I am syncing with Pulp3. I noticed that pulpcore-workers goes up to 2.7g of virtual memory. Installer set 4 of them for my system and luckily only two of them were working when I was syncing my 4 repositories, but that is significant spike from the previous state.
My suggestion would be to verify how installer sets things up for multicore systems, e.g. systems with many cores could suffer from increased memory usage when syncing.
I know this thread is a little older now, but I really wanted to share a summary of my setup and the common resource usage, in the case it may still be useful to this community.
The below is my Foreman and Katello version. I know at this point, these are already “old”. My challenge currently is that I’m in the middle of a large deployment of CentOS to 5000+ hosts in a retail environment. It is a very slow deployment (such is the nature of retail), so I’m trying to not do any upgrades until completed.
).
)