Metrics for better tuning presets

[evgeni]

Ohai,

as you may have seen, there have been a few discussions around the memory usage of Foreman (and related parts of the stack):

Foreman install 2.5, HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed

We are looking to upgrade to foreman 2.5. By upgrade I don’t mean actually updating the packages, but installing a new 2.5 instance. Previously we where using passenger with Foreman 2.2 with no issues. With 2.5 I’ve been unable to get the webserver up and running. The logs show: [Wed Sep 22 20:03:04.306313 2021] [proxy:error] [pid 281] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed [Wed Sep 22 20:03:04.306376 2021] [proxy:error] [pid 281] AH00959…

Foreman 3.0 and Katello 4.2 memory issue

Problem: When i used Foreman 2.5 and Katello 4.1.3 i had no memory issue. It was very stable. After i updated Katello 4.1.4 the memory problem began. It went up and i was getting messages: "Linux: OOM killed a process. Out of memory: Kill process 2979 (ruby). So i decided to upgrade Foreman and Katello to the latest version and hoped that it will solve my issues. But unfortunately still memory usage is increasing and its not stable. Here is a memory usage picture after the upgrade: [memory u…

Foreman 2.5.3 performance issues (Memory loss, timeouts)

This thread: Foreman install 2.5, HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed started getting lost on these issues so creating a new thread to report on performance issues we are seeing with Foreman 2.5.3. After some troubles getting Foreman 2.5.3 running (see linked thread) we’ve seen dramatic memory usage compared to 2.2. We’ve upped the memory on a couple of VMs running Foreman in various environments without much change. I’ve attached some performance graphs on what we are …

Increased memory on 2.5 with smart class params

Hey, I was trying to help a user with increased memory after upgrade to 2.5 and we looked into typical offenders like NICs or facts. But recent patches did not solve the problem, it looks like there are hosts with many (900) puppet smart class params overrides: https://projects.theforeman.org/issues/33585 Can someone familiar with the codebase look into this? There might be a memory leak introduced recently. Thanks!

While we think we have fixed the source for the particular increased memory usage in 2.5/3.0, we also found that our current tuning defaults for deployments aren’t as well defined as they could. This is mostly due to the fact that we have ways to do calculations of the form of “if you have 4 CPUs and 32GB of memory, you shouldn’t run more than 6 Puma workers” these calculations aren’t based on recent numbers and platform changes and might be suboptimal.

Let me quickly recap the current architecture:

Foreman

• Apache HTTPd: Doing, well, HTTP. So SSL termination, forwarding requests to backends using mod_proxy, no mod_wsgi or mod_passenger anymore
• Puma: The Ruby application server, running the main Ruby on Rails app.
• PostgreSQL: The database.
• Dynflow: The “Foreman jobs daemon”, doing various background tasks that the Rails app shouldn’t be blocked on.
• Redis: Mostly used as a cache for Dynflow and Rails

Foreman Proxy

• Foreman Proxy: Small Sinatra web app, running on WEBRick, responsible for integrating with external services like DNS, DHCP, Puppet etc.

Puppet Server

• Puppet Server: If you’re using Puppet, you’ll also have a Puppet Server running on your machine

Katello

If you’re using Katello, you’ll also have the following additional services running on your system:

• Pulpcore API: Django app (running in gunicorn) allows Foreman/Katello to manage Pulp repositories
• Pulpcore Content: Django app (running in gunicorn) serves content (packages etc) to clients
• Pulpcore Workers: Django app (running standalone) that does the “work” of downloading packages, generating repository metadata etc
• Tomcat: Java application server, running the Candlepin app, responsible for all things subscriptions for Katello and Clients

Now, as you can see, depending on the deployed featureset, the number of services (and thus required resources) greatly differ. Additionally, y’all will have different use cases and use patterns, resulting in different resource usage, even with the same features enabled.

What we’re looking for are numbers to further improve our defaults. Mostly memory and CPU usage of the individual processes, plus a bit of details about your deployment.

Examples:

• Foreman without Katello and Puppet on 4 CPU, 16GB RAM VM. 6 Puma workers at 1GB each, PostgreSQL 800MB, Redis 200MB, running flawlessly
• Foreman with Katello and Puppet on 8 CPU, 32GB RAM. 12 Puma workers at 2GB each, PostgreSQL at 3GB, Redis at 1GB, Tomcat at 2GB and Puppet Server at 4GB, constantly OOMing