Well, I added the excludes I have to both “Ignore interfaces with matching identifier” and also “Exclude pattern for facts stored in foreman”, restarted apache and started to monitor RAM usage like before, and here it is:
I´d say it helped a little with average usage but did nothing for when the two hypervisors where running. As you can see, the usage spiked and fell again after the next 5 minute interval for when ‘passenger-recycler’ kicked in. Had I disabled ‘passenger-recycler’, I expect the graph to go have gone as rampant as the last one.
This still smells to me like something with the hypervisor fact processing causing the extra load. Perhaps they have a large number of block devices? Would you be able to provide us with the facter output from the hypervisors?
Ok, so it doesn’t seem like block devices. There are 16 and 19 interfaces on them, could you share the setting you have in place for Ignore interfaces with matching identifier? maybe something there is wrong leading to some of the interfaces still being created.
Adding interface filter is not enough, you also need to delete rogue records from your database. Our code keeps loading them into memory every report, that’s what’s make your memory consumption high. There is a rake task that does the job for you, recently it has been refactored to take filters into account:
You only need to run this task if you change the ignored interface list, to ensure there aren’t any leftover interfaces. Out of curiosity, how long did it take to clean all of those interfaces?
I can hear the fat lady singing guys! So I left it for reeeally long sample and look:
And this is with ‘passenger-recycler’ disabled to! Also, the load on the server has gone from like a steady three to like a point five or so, very nice!
So I think we can mark this case as solved, with a big fat red sticker that says: “Make damned sure to disable unnecessary network interfaces!”
Thank you so much guys, I´ve learned so much about how to debug this application and how the memory management system works (and not)
Is this something you have to do regularly or just this once?
