Foreman application server (Puma) status monitoring

Tutorials
1

This post describes steps to monitor and control the puma service in production environment on RPM based distribution. It is working with Foreman 3.0 and newer. Tested on CentOS 7 and CentOS 8 Stream distributions. On CentOS 7 you need to wrap up commands in scl enable tfm '$command'.

Using pumactl for gettings stats or gc-stats

# RHEL 7 like system
scl enable tfm 'pumactl stats -S /usr/share/foreman/tmp/puma.state'
# RHEL 8 / CentOS 8 Stream like
pumactl stats -S /usr/share/foreman/tmp/puma.state
pumactl gc-stats -S /usr/share/foreman/tmp/puma.state

example outpus:

stats

{
   "started_at":"2021-09-02T07:39:16Z",
   "workers":2,
   "phase":0,
   "booted_workers":2,
   "old_workers":0,
   "worker_status":[
      {
         "started_at":"2021-09-02T07:39:16Z",
         "pid":10473,
         "index":0,
         "phase":0,
         "booted":true,
         "last_checkin":"2021-09-23T10:44:19Z",
         "last_status":{
            "backlog":0,
            "running":0,
            "pool_capacity":16,
            "max_threads":16,
            "requests_count":5582
         }
      },
      {
         "started_at":"2021-09-02T07:39:16Z",
         "pid":10476,
         "index":1,
         "phase":0,
         "booted":true,
         "last_checkin":"2021-09-23T10:44:17Z",
         "last_status":{
            "backlog":0,
            "running":0,
            "pool_capacity":16,
            "max_threads":16,
            "requests_count":5282
         }
      }
   ]
}

gc-stats

{
   "count":148,
   "heap_allocated_pages":5365,
   "heap_sorted_length":5365,
   "heap_allocatable_pages":0,
   "heap_available_slots":2186774,
   "heap_live_slots":2186537,
   "heap_free_slots":237,
   "heap_final_slots":0,
   "heap_marked_slots":1568935,
   "heap_eden_pages":5365,
   "heap_tomb_pages":0,
   "total_allocated_pages":5365,
   "total_freed_pages":0,
   "total_allocated_objects":49608573,
   "total_freed_objects":47422036,
   "malloc_increase_bytes":71808,
   "malloc_increase_bytes_limit":16777216,
   "minor_gc_count":128,
   "major_gc_count":20,
   "compact_count":0,
   "remembered_wb_unprotected_objects":37228,
   "remembered_wb_unprotected_objects_limit":74036,
   "old_objects":1477246,
   "old_objects_limit":2785436,
   "oldmalloc_increase_bytes":82064,
   "oldmalloc_increase_bytes_limit":62801294
}

To display puma user friendly status, run this

/usr/sbin/foreman-puma-status

this gives you info about the CPU and memory usage, number of requests, current load

status1
status11405×72 17.6 KB

It works on both 7 and 8 Stream. It does not have execution permission by default, that’s why you need to add sh to the command.

For continuous monitoring, you can use watch like this

watch --color --interval 1 sh ~foreman/script/foreman-puma-status

For the historical reasons I’m also keeping pre 3.0 notes which may work to some degree

For version with Foreman older than 3.0 see previous versions of this wiki

3 Likes
2

In Fixes #29507 - Add puma-status for reporting by ehelms · Pull Request #7735 · theforeman/foreman · GitHub we force the control app path which should avoid the private tmp problem. Isn’t that an easier solution?

1 Like
3

Thanks for linking the PR, once it’s merged I’ll update the post (I made it a wiki already)

4

As puma is already using sd_notify, would it make sense to have this always in systemctl status output? If yes, I think an issue or pull-request would be considered upstream.

5

The puma-status gem is extremely useful and it is pretty simple, I wonder if we could integrate this onto our About page :slight_smile:

Nice writeup.

6

I do think it would be considered. It was in GitHub - sj26/puma-plugin-systemd: Puma integration with systemd for better daemonising under modern Linux systemds: notify, status, watchdog but that plugin kept getting out of sync so I finished a PR by another contributor that at least gave the minimal parts we needed. Implementing what you suggest is a good idea.

IMHO about is the wrong place. If you have a load balanced Foreman, there is just information about the current host.

7

I created Utilize systemd-notify to add status details · Issue #2604 · puma/puma · GitHub, so we will see what happens.

8

Note that what was originally added to the puma-plugin-systemd for systemctl output was basic worker and thread count information. The puma-status gem displays more data such as memory and load. If they’d accept all things puma-status shows to be displayed there it would be nice all in one solution. Until then, we have one week till stabilization so if we’d like puma-status reporting (which I would like) reviews of Fixes #29507 - Add puma-status for reporting by ehelms · Pull Request #7735 · theforeman/foreman · GitHub would be appreciated so I can also get the packaging part done.

9

I’ve updated the wiki with much easier use now, also with example outputs.

2 Likes
10

Should it use /usr/sbin/foreman-puma-status instead of sh ~foreman/script/foreman-puma-status? That’s what we actually support and should also work on Debian.

11

Why we ship the script then? That seems to correctly handle SCL on el7.

12

I think it is an oversight. We first patch the script here:

Then we copy it to the final location:

However, we then don’t exclude the actual script. Looking at it, there may be more scripts that are shipped twice or could be excluded.

13

ok, thanks for sharing, updated

14

There was a pull request for the systemd status, I tested it already successfully and pinged some of you on github so you can provide feedback there. If someone else wants to see how it would look or add suggestions: Utilize systemd-notify to add status details · Issue #2604 · puma/puma · GitHub