Foreman seems to crash often because it cannot fork new processes because it cannot allocate memory (errno=12)
Should not be there
Foreman and Proxy versions:
I’m running on 2.1.4 (just upgraded) but on 2.1.3 was the same
Foreman and Proxy plugin versions:
Distribution and version:
Other relevant data:
My first thought would be, out of memory (OS level). But everytime I check free -m there should be enough free memory.
One thing to add, might be coincidence, but the problems started after this machine was updated to RHEL 7.9! As I cannot find records before this upgrade!
This is the last part of the log, the first 100 lines are all the same: The cannot fork new process message.
[ 2020-10-29 13:42:08.0453 14310/7f6d64457700 Pool2/Pool.h:760 ]: ERROR: Cannot fork() a new process: Cannot allocate memory (errno=12)
in ‘long long unsigned int Passenger::ApplicationPool2::Pool::realCollectAnalytics()’ (Pool.h:867)
in ‘static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)’ (Pool.h:754)
terminate called after throwing an instance of ‘boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorboost::thread_resource_error >’
what(): boost::thread_resource_error: Resource temporarily unavailable
ERROR: cannot fork a process for executing ‘tee’
[ pid=14310, timestamp=1603975328 ] Process aborted! signo=SIGABRT(6), reason=SI_TKILL, signal sent by PID 14310 with UID 0, si_addr=0x37d6, randomSeed=1603975300
[ pid=14310 ] Could not create crash log file, so dumping to stderr only.
[ pid=14310 ] Could fork a child process for dumping diagnostics: fork() failed with errno=12
[ 2020-10-29 13:42:08.0600 13868/7fbae9320700 agents/Watchdog/AgentWatcher.cpp:96 ]: Phusion Passenger helper agent (pid=14310) crashed with signal SIGABRT, restarting it…