Setting up Prometheus Metrics / Grafana for Foreman Performance Monitoring

Support
1

I am trying to figure out what all the metrics mean that Im getting from Foreman. Its awesome there are so many of them. Ive spent quite a while with various AI overlords asking them what each of these mean. Trying to understand it all. I do understand what some of it is, but I’m not sure I understand what to really look for, in regards to performance monitoring.

The list is vast. I understand some things like http requests and how an increasing amount of them would affect performance. But Im failing miserably to understand how garbage collection affects performance, what sort of graphs and metrics to even look at based on so much of this amazing data.

Has anyone worked closely with the prometheus data exported by Foreman, and if so, what are you looking at to determine performance?

Thanks.

2

I think on their own they are quite useful, especially if you’re in a very modular environment where you separate the components onto individual nodes.

For me their main value is the performance of the ruby environment monitoring, but as lzap pointed out in this threads, it’s more stable to use statsd

I"ve not used the co-pilot monitoring here so can’t comment on it’s value, I’m sure it’s good though
https://docs.theforeman.org/3.16/Monitoring_Project/index-foreman-el.html

I’ve found the best way to get a picture (because most of my environments are now invested in prometheus and time based stats) is to run the apache exporter, the postgress exporter, the puppet-operational-dashboard (and exporter) and use a few instance seperations to get a clear end to end view or foremans web components, it’s ruby internals, it’s database, and the cost and performance of puppet (using additional exporters like memcache etc) but pulling the individual components and summing them into a single grafana dashboard has worked well for me so far.

2 Likes
3

I swapped over to statsd, but Im not quite understanding its config.
Im seeing:

2025-09-15T17:28:27 [E|app|764af62a] [StatsD] Errno::ECONNREFUSED: Connection refused - send(2)
2025-09-15T17:28:27 [E|app|764af62a] [StatsD] Errno::ECONNREFUSED: Connection refused - send(2)
2025-09-15T17:28:27 [E|app|764af62a] [StatsD] Errno::ECONNREFUSED: Connection refused - send(2)

In the logs. Here is my config:

:telemetry:
  # prefix for all metrics
  :prefix: 'fm_rails'
  # prometheus endpoint is at /metrics
  # warning: ruby client library currently does not supprt multi-process web servers
  :prometheus:
    :enabled: false
  # works with statsd_exporter too, use the rake task to generate config
  :statsd:
    :enabled: true
    # IP and port (do not use DNS)
    :host: '127.0.0.1:8125'
    # one of 'statsd', 'statsite' or 'datadog'
    :protocol: 'statsd'
  # Rails logs end up in logger named 'telemetry' when enabled
  :logger:
    :enabled: false
    # logging level as in Logger::LEVEL
    :level: 'DEBUG'

So just basic vanilla. While enabling prometheus everything worked fine, I saw metrics immediately. With stats, I cannot get anything to show up.
Im not really sure what sort of connection is being refused.

There is little to no documentation or public discussion about this, so hoping someone here knows.

5

Hello,

you need to run a statsd daemon on the port 8125, note this is a UDP protocol not TCP one. If you are on Fedora/EL system, you can take advantage of the pcp-statsd agend that is available which forwards data to the PCP. Foreman installer is able to setup pcp automatically I think.

Or run any other statsd collector available. If you are using Prometheus, then you can use this:

6

Thanks @lzap - I got it working. The issue we were running into was a bad port setting inside the alloy config. All is now working without using the statsd binary, just using the built in alloy statsd prometheus scraper.