Monitoring Foreman metrics

[jtruestedt]

Hi,

I am currently searching for the “right” way to monitor metrics of Foreman. However, I found two guides here:

• Monitoring Foreman with Prometheus
• Monitoring Foreman with Prometheus via statsd

So the first one is from 2019 but the last post of 2022 says:

@lzap

Just use statsd and everything is stable, the Ruby prometheus client library hasn’t been fixed. It is just too much to handle for temporary files.

What I found regarding that ruby-prometheus-client is slow response on metrics http endpoint · Issue #194 · prometheus/client_ruby · GitHub, which is still open (with the question if it is fixed or not but there is no definite answer)

The second one is from 2020 is the mentioned statsd way to do it.
Also this post looks like I should use statsd: How to organize Foreman monitoring? - #4 by Marek_Hulan

@Marek_Hulan

It sounds you’re interested in performance monitoring. See if this helps. Here’s the deep dive recording. It’s a bit dated and I think @ehelms planned to update that and perhaps include grafana, not sure if there’s some progress already. There’s also this tutorial.

If you’re more interested in Foreman’s health, e.g. how many tasks are stuck or how many hosts are in red state, it was discussed at this thread Monitoring Foreman with Prometheus via statsd .

Hope that helps.

And this is basically the state of beginning 2023 - as far as I found discussions on that topic here in the community-forum. So just following that, I would guess I should use statsd.

But there is also the docs part:
https://docs.theforeman.org/nightly/Monitoring_Project/index-foreman-el.html
which is explaining the first way using the ruby-client of prometheus.
This documentation is also part of the documentation for RedHats Satellite 6.15 (Chapter 3. Setting up the metrics monitoring solution | Red Hat Product Documentation).

So that sounds that it might be fixed and now the prometheus client should be the way to go?

Is anyone monitoring foreman metrics with one of those solutions and maybe can confirm if that ruby-client-issue for prometheus is gone?

[evgeni]

The prometheus client isn’t “fixed” as such, but we adjusted our usage of it to not run into the described problems too much.

So yes, the docs link is correct, please use it and report issues you find with it

[lzap]

By the way statsd made it into RHEL and is able to collect a lot of data without a hiccup. @pmoravec found an issue this month but it was quickly found and fixed in pcp-statsd so latest update should solve it too.