Metrics for better tuning presets

Ohai,

as you may have seen, there have been a few discussions around the memory usage of Foreman (and related parts of the stack):

While we think we have fixed the source for the particular increased memory usage in 2.5/3.0, we also found that our current tuning defaults for deployments aren’t as well defined as they could. This is mostly due to the fact that we have ways to do calculations of the form of “if you have 4 CPUs and 32GB of memory, you shouldn’t run more than 6 Puma workers” these calculations aren’t based on recent numbers and platform changes and might be suboptimal.

Let me quickly recap the current architecture:

Foreman

  • Apache HTTPd: Doing, well, HTTP. So SSL termination, forwarding requests to backends using mod_proxy, no mod_wsgi or mod_passenger anymore
  • Puma: The Ruby application server, running the main Ruby on Rails app.
  • PostgreSQL: The database.
  • Dynflow: The “Foreman jobs daemon”, doing various background tasks that the Rails app shouldn’t be blocked on.
  • Redis: Mostly used as a cache for Dynflow and Rails

Foreman Proxy

  • Foreman Proxy: Small Sinatra web app, running on WEBRick, responsible for integrating with external services like DNS, DHCP, Puppet etc.

Puppet Server

  • Puppet Server: If you’re using Puppet, you’ll also have a Puppet Server running on your machine

Katello

If you’re using Katello, you’ll also have the following additional services running on your system:

  • Pulpcore API: Django app (running in gunicorn) allows Foreman/Katello to manage Pulp repositories
  • Pulpcore Content: Django app (running in gunicorn) serves content (packages etc) to clients
  • Pulpcore Workers: Django app (running standalone) that does the “work” of downloading packages, generating repository metadata etc
  • Tomcat: Java application server, running the Candlepin app, responsible for all things subscriptions for Katello and Clients

Now, as you can see, depending on the deployed featureset, the number of services (and thus required resources) greatly differ. Additionally, y’all will have different use cases and use patterns, resulting in different resource usage, even with the same features enabled.

What we’re looking for are numbers to further improve our defaults. Mostly memory and CPU usage of the individual processes, plus a bit of details about your deployment.

Examples:

  • Foreman without Katello and Puppet on 4 CPU, 16GB RAM VM. 6 Puma workers at 1GB each, PostgreSQL 800MB, Redis 200MB, running flawlessly
  • Foreman with Katello and Puppet on 8 CPU, 32GB RAM. 12 Puma workers at 2GB each, PostgreSQL at 3GB, Redis at 1GB, Tomcat at 2GB and Puppet Server at 4GB, constantly OOMing
3 Likes