HI all,
My foreman server has developed a performance problem seemingly overnight,
and after spending the day trying to unravel it, I'm looking for
suggestions. I'm running Foreman 1.7.2-1, Puppet 3.6.2,-1, PuppetDB
2.2.2-1, and PosgreSQL 8.4.20-1 on CentOS 6.6. The last yum update was
done on April 30th. The system has been running well for a long time. I
realize this might be a pure puppet issue, but since I've used the foreman
installer for everything, I thought I'd start here.
Basically every 30 minutes for about 5 minutes the load average on the
foreman server spikes hard enough to have anywhere from 10-25 hosts error
on puppet run with a 503 This website is under heavy load message in the
puppet report. This began about 9PM localtime last night and has been
going on ever since.
I have investigated the following areas:
- Restarted Apache/Passenger (foreman and puppetmaster), puppetdb, and
postgresql. - Restarted the entire server.
- Trend Counters: I removed the rake task from the /etc/cron.d/foreman
cron script. - Adjusted the PassengerMaxPoolSize from 12 to 16
- Set PassengerMinInstances to 4
- Set PassengerMaxRequests to 1000
- Configured my Nagios server (uses exported resources for about 210
hosts and 2100 services) from running at 5 and 35 after the hour to 17 and
47 after the hour…well outside of the event window. The nagios server
has been running puppet out of cron for a very long time, I just adjusted
the execution time here thinking the resource collection might be colliding
with something else. - Watched log files across all components - to find nothing useful
- Observed top finding Passenger Puppet processes running and stacking
up, but i'm not able to exactly determine which puppet agent hosts are tied
to which process that seem to be stacking up.
I've attached a PNP4nagios graphic showing the load average pattern in the
short run and one for the long run.
Any ideas would be greatly appreciated. Many thanks!
