Puma status monitoring - not working at the moment

This post describes steps to to enable monitoring and controlling of the puma service in production environment on RPM based distribution. Currently it’s not working for various reasons, but with newer versions of puma, it should, so dumping here for the future use.

Using pumactl

  1. enable the control socket for puma protected with the token
echo "activate_control_app\nstate_path '/run/foreman/puma.stats'" >> ~foreman/config/puma/production.rb

note in development environment, you can add this to config/puma.rb (new file)

  1. restart the puma server
systemctl restart foreman
  1. read the token and url from the stats file, find the private tmp directory for the Foreman service
cat /run/foreman/puma.stats
ls -d /tmp/systemd-private-*foreman*

given the Foreman’s tmp is using private tmp directory, you must construct the unix socket URL manually, take the unix:// + private_tmp + control_url without the unix:// prefix

so for example the final url could be unix:///tmp/systemd-private-70696e8a3a6748d197fe09749efc0bc3-foreman.service-PjLEQU/tmp/puma-status-1618483682415-26762

this URL with the token needs to be specified for each pumactl command

  1. You can now use the pumactl commandline utility, namely stats, gc-stats can be useful. On RPM-based production setups, one can run it by
scl enable tfm 'pumactl stats --control-url unix:///tmp/systemd-private-70696e8a3a6748d197fe09749efc0bc3-foreman.service-PjLEQU/tmp/puma-status-1618483682415-26762 --control-token 12ff5f4af3e28b509774ffeb4b7c7790'

Today, you’d receive too long unix socket path (112bytes given but 108bytes max) error, but with newer Puma than we ship today, it seems to work fine.

puma-status, monitoring the load

There’s also a puma-status gem one can install, pointing it to the /run/foreman/puma.stats shows currect status and load of workers. It requires 2.4, so you’d need to install it into SCL (not recommended, only if you know what you’re doing). It may be watched like this

watch --interval 0.1 --color scl enable tfm 'puma-status /run/foreman/puma.stats'

but it fails to correctly find the private tmp directory and there’s no way to specify the control socket directly. The only workaround I found so far was to copy the /run/foreman/puma.stats to another file and manually update the control_url there. Then again, it fails with too long unix socket path (112bytes given but 108bytes max)

The output you’d normally see looks like this

1643714 (/tmp/puma.stats) Uptime:  0m16s | Load: 0[     ]16 | Req: 0
 └ 1643714 CPU:   0.0% Mem:  344 MB Uptime:  0m16s | Load: 0[     ]16 | Req: 0

watching it can tell you if you’re using all 16 threads and whether increasing it’s maximum could help improving the performance.

3 Likes

In Fixes #29507 - Add puma-status for reporting by ehelms · Pull Request #7735 · theforeman/foreman · GitHub we force the control app path which should avoid the private tmp problem. Isn’t that an easier solution?

1 Like

Thanks for linking the PR, once it’s merged I’ll update the post (I made it a wiki already)

As puma is already using sd_notify, would it make sense to have this always in systemctl status output? If yes, I think an issue or pull-request would be considered upstream.

The puma-status gem is extremely useful and it is pretty simple, I wonder if we could integrate this onto our About page :slight_smile:

Nice writeup.

I do think it would be considered. It was in GitHub - sj26/puma-plugin-systemd: Puma integration with systemd for better daemonising under modern Linux systemds: notify, status, watchdog but that plugin kept getting out of sync so I finished a PR by another contributor that at least gave the minimal parts we needed. Implementing what you suggest is a good idea.

IMHO about is the wrong place. If you have a load balanced Foreman, there is just information about the current host.

I created Utilize systemd-notify to add status details · Issue #2604 · puma/puma · GitHub, so we will see what happens.

Note that what was originally added to the puma-plugin-systemd for systemctl output was basic worker and thread count information. The puma-status gem displays more data such as memory and load. If they’d accept all things puma-status shows to be displayed there it would be nice all in one solution. Until then, we have one week till stabilization so if we’d like puma-status reporting (which I would like) reviews of Fixes #29507 - Add puma-status for reporting by ehelms · Pull Request #7735 · theforeman/foreman · GitHub would be appreciated so I can also get the packaging part done.