I'd like to get an idea of why our production Katello server barfs all over
itself semi-regularly.
What we see is that in the 'Run Distribution' graph on the Dashboard there
would be about 8 runs with an average of ~50 nodes running puppet
successfully. Then there's always the one that spikes to 210, then they all
go back to ~50 for 8 or 9 runs then it spikes to ~210 again fairly
regularly. So far what I've found is that passenger-status looks like the
below snippet when the spike occurs.
On the Web-GUI and log side this show up as 'Server Busy' errors and failed
puppet runs. My question is does anyone have an idea that would help me
find what task or locked resource is causing this behaviour?
CentOS 7.3
48 Core, 64 GB RAM
[root@katello3 emanners]# passenger-status
Version : 4.0.18
Instance: 42454
----------- General information -----------
Max pool size : 48
Processes : 48
Requests in top-level queue : 0
----------- Application groups -----------
/usr/share/foreman#default:
App root: /usr/share/foreman
Requests in queue: 17
- PID: 9748 Sessions: 1 Processed: 4480 Uptime: 1h 40m 50s
CPU: 8% Memory : 752M Last used: 1s ago - PID: 14732 Sessions: 1 Processed: 3445 Uptime: 1h 30m 50s
CPU: 8% Memory : 646M Last used: 1s ago - PID: 26952 Sessions: 1 Processed: 2178 Uptime: 1h 5m 51s
CPU: 7% Memory : 719M Last used: 22s ago - PID: 47952 Sessions: 1 Processed: 2039 Uptime: 40m 48s
CPU: 11% Memory : 565M Last used: 1s ago - PID: 48023 Sessions: 1 Processed: 1656 Uptime: 40m 46s
CPU: 10% Memory : 578M Last used: 19s ago - PID: 55165 Sessions: 1 Processed: 797 Uptime: 25m 47s
CPU: 8% Memory : 571M Last used: 1s ago - PID: 60497 Sessions: 1 Processed: 649 Uptime: 15m 54s
CPU: 13% Memory : 530M Last used: 1s ago - PID: 60528 Sessions: 1 Processed: 1009 Uptime: 15m 54s
CPU: 18% Memory : 541M Last used: 1s ago - PID: 64833 Sessions: 1 Processed: 581 Uptime: 8m 4s
CPU: 23% Memory : 557M Last used: 1s ago
/etc/puppet/rack#default:
App root: /etc/puppet/rack
Requests in queue: 0
- PID: 37425 Sessions: 1 Processed: 10706 Uptime: 8h 2m 4s
CPU: 3% Memory : 95M Last used: 4s ago - PID: 44806 Sessions: 1 Processed: 4551 Uptime: 6h 2m 4s
CPU: 2% Memory : 92M Last used: 10s ago - PID: 51235 Sessions: 1 Processed: 5094 Uptime: 4h 2m 5s
CPU: 3% Memory : 94M Last used: 0s ago - PID: 21495 Sessions: 0 Processed: 4348 Uptime: 3h 2m 4s
CPU: 4% Memory : 93M Last used: 0s ago - PID: 21563 Sessions: 1 Processed: 1919 Uptime: 3h 2m 4s
CPU: 1% Memory : 85M Last used: 1s ago - PID: 21711 Sessions: 1 Processed: 5769 Uptime: 3h 2m 4s
CPU: 5% Memory : 78M Last used: 0s ago - PID: 58352 Sessions: 1 Processed: 4484 Uptime: 2h 2m 4s
CPU: 6% Memory : 94M Last used: 7s ago - PID: 58402 Sessions: 1 Processed: 2226 Uptime: 2h 2m 4s
CPU: 3% Memory : 83M Last used: 41s ago - PID: 58446 Sessions: 1 Processed: 1324 Uptime: 2h 2m 4s
CPU: 1% Memory : 82M Last used: 2s ago - PID: 58491 Sessions: 1 Processed: 3400 Uptime: 2h 2m 4s
CPU: 4% Memory : 81M Last used: 5s ago - PID: 58520 Sessions: 0 Processed: 1131 Uptime: 2h 2m 4s
CPU: 1% Memory : 81M Last used: 0s ago - PID: 58534 Sessions: 1 Processed: 1325 Uptime: 2h 2m 4s
CPU: 1% Memory : 82M Last used: 8s ago - PID: 58555 Sessions: 0 Processed: 2774 Uptime: 2h 2m 4s
CPU: 3% Memory : 83M Last used: 5s ago - PID: 28766 Sessions: 1 Processed: 931 Uptime: 1h 2m 4s
CPU: 2% Memory : 81M Last used: 4s ago - PID: 28774 Sessions: 0 Processed: 893 Uptime: 1h 2m 4s
CPU: 2% Memory : 81M Last used: 0s ago - PID: 28785 Sessions: 1 Processed: 2817 Uptime: 1h 2m 4s
CPU: 8% Memory : 134M Last used: 4s ago - PID: 28807 Sessions: 1 Processed: 1206 Uptime: 1h 2m 4s
CPU: 3% Memory : 84M Last used: 2s ago - PID: 28871 Sessions: 1 Processed: 558 Uptime: 1h 2m 4s
CPU: 1% Memory : 77M Last used: 5s ago - PID: 28895 Sessions: 1 Processed: 4121 Uptime: 1h 2m 4s
CPU: 12% Memory : 83M Last used: 11s ago - PID: 28921 Sessions: 1 Processed: 2250 Uptime: 1h 2m 4s
CPU: 6% Memory : 82M Last used: 7s ago - PID: 28941 Sessions: 1 Processed: 1699 Uptime: 1h 2m 4s
CPU: 4% Memory : 79M Last used: 4s ago - PID: 28967 Sessions: 1 Processed: 2939 Uptime: 1h 2m 4s
CPU: 8% Memory : 81M Last used: 9s ago - PID: 28980 Sessions: 1 Processed: 2499 Uptime: 1h 2m 4s
CPU: 6% Memory : 81M Last used: 3s ago - PID: 29002 Sessions: 1 Processed: 787 Uptime: 1h 2m 4s
CPU: 2% Memory : 81M Last used: 46s ago - PID: 2808 Sessions: 1 Processed: 267 Uptime: 2m 5s
CPU: 24% Memory : 71M Last used: 5s ago - PID: 2817 Sessions: 1 Processed: 170 Uptime: 2m 5s
CPU: 20% Memory : 83M Last used: 8s ago - PID: 2831 Sessions: 0 Processed: 168 Uptime: 2m 5s
CPU: 21% Memory : 71M Last used: 15s ago - PID: 2842 Sessions: 1 Processed: 239 Uptime: 2m 5s
CPU: 21% Memory : 76M Last used: 2s ago - PID: 2853 Sessions: 0 Processed: 405 Uptime: 2m 5s
CPU: 30% Memory : 72M Last used: 1s ago - PID: 2866 Sessions: 1 Processed: 293 Uptime: 2m 5s
CPU: 25% Memory : 72M Last used: 0s ago - PID: 2877 Sessions: 1 Processed: 185 Uptime: 2m 5s
CPU: 23% Memory : 79M Last used: 4s ago - PID: 2889 Sessions: 1 Processed: 378 Uptime: 2m 5s
CPU: 31% Memory : 71M Last used: 4s ago - PID: 2901 Sessions: 1 Processed: 296 Uptime: 2m 5s
CPU: 26% Memory : 72M Last used: 5s ago - PID: 2913 Sessions: 1 Processed: 175 Uptime: 2m 5s
CPU: 19% Memory : 80M Last used: 2s ago - PID: 2925 Sessions: 1 Processed: 339 Uptime: 2m 4s
CPU: 28% Memory : 70M Last used: 3s ago - PID: 2936 Sessions: 1 Processed: 299 Uptime: 2m 4s
CPU: 26% Memory : 76M Last used: 4s ago - PID: 2953 Sessions: 1 Processed: 173 Uptime: 2m 4s
CPU: 19% Memory : 75M Last used: 0s ago - PID: 2967 Sessions: 1 Processed: 302 Uptime: 2m 4s
CPU: 25% Memory : 81M Last used: 2s ago - PID: 2979 Sessions: 1 Processed: 229 Uptime: 2m 4s
CPU: 23% Memory : 71M Last used: 1s ago