I can hear the fat lady singing guys! So I left it for reeeally long sample and look:
And this is with ‘passenger-recycler’ disabled to! Also, the load on the server has gone from like a steady three to like a point five or so, very nice!
So I think we can mark this case as solved, with a big fat red sticker that says: “Make damned sure to disable unnecessary network interfaces!”
Thank you so much guys, I´ve learned so much about how to debug this application and how the memory management system works (and not) 
/K