Foreman server crashes when the load is around 150

Royee_Tager · July 17, 2014, 2:29am

Hello,

Lately my Foreman server crashes.
The top command shows that the load is 150 and I have 6 zombies.

The Apache error_log shows some strange error:

[ 2014-07-17 05:02:22.6186 28747/7f5909400720 agents/Watchdog/Main.cpp:631
]: All Phusion Passenger agents started!
[Thu Jul 17 05:02:22 2014] [notice] Digest: generating secret for digest
authentication …
[Thu Jul 17 05:02:22 2014] [notice] Digest: done
[ 2014-07-17 05:02:22.6994 28769/7ffe47dac720 agents/Watchdog/Main.cpp:450
]: Options: { 'analytics_log_user' => 'nobody', 'default_group' =>
'nobody', 'default_python' => 'python', 'default_ruby' => 'ruby',
'default_user' => 'nobody', 'log_level' => '0', 'max_instances_per_app' =>
'0', 'max_pool_size' => '50', 'passenger_root' =>
'/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini',
'pool_idle_time' => '1500', 'temp_dir' => '/tmp',
'union_station_gateway_address' => 'gateway.unionstationapp.com',
'union_station_gateway_port' => '443', 'user_switching' => 'true',
'web_server_pid' => '28767', 'web_server_type' => 'apache',
'web_server_worker_gid' => '48', 'web_server_worker_uid' => '48' }
[ 2014-07-17 05:02:22.7038 28772/7f047e74b720
agents/HelperAgent/Main.cpp:602 ]: PassengerHelperAgent online, listening
at unix:/tmp/passenger.1.0.28767/generation-0/request
[ 2014-07-17 05:02:22.7072 28778/7f8f0f2897e0
agents/LoggingAgent/Main.cpp:318 ]: PassengerLoggingAgent online, listening
at unix:/tmp/passenger.1.0.28767/generation-0/logging
[ 2014-07-17 05:02:22.7075 28769/7ffe47dac720 agents/Watchdog/Main.cpp:631
]: All Phusion Passenger agents started!
[Thu Jul 17 05:02:22 2014] [notice] Apache/2.2.15 (Unix) DAV/2
Phusion_Passenger/4.0.18 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured –
resuming normal operations
[ 2014-07-17 05:02:24.2545 28772/7f047c852700 Pool2/Spawner.h:738 ]: [App
28802 stdout]
[ 2014-07-17 05:02:25.1315 28772/7f047c852700 Pool2/SmartSpawner.h:301 ]:
Preloader for /etc/puppet/rack started on PID 28802, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.28802
[ 2014-07-17 05:02:26.2069 28772/7f047c566700 Pool2/Spawner.h:738 ]: [App
29081 stdout]
[ 2014-07-17 05:02:43.1300 28772/7f047c566700 Pool2/SmartSpawner.h:301 ]:
Preloader for /usr/share/foreman started on PID 29081, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.29094
[ 2014-07-17 05:09:07.6952 28772/7f047c852700 Pool2/Spawner.h:738 ]: [App
4599 stdout]
[ 2014-07-17 05:09:29.3475 28772/7f047c852700 Pool2/SmartSpawner.h:301 ]:
Preloader for /usr/share/foreman started on PID 4599, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.4632
[ 2014-07-17 05:19:07.8990 28772/7f047c437700 Pool2/Spawner.h:738 ]: [App
17278 stdout]
[ 2014-07-17 05:19:34.6716 28772/7f047c437700 Pool2/SmartSpawner.h:301 ]:
Preloader for /usr/share/foreman started on PID 17278, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.17306

Thanks,

Royee

lzap · July 17, 2014, 2:50pm

Hi,

Foreman version? OS version? Passenger version?

LZ

···

On Wed, Jul 16, 2014 at 07:29:44PM -0700, Royee Tager wrote: > Hello, > > Lately my Foreman server crashes. > The top command shows that the load is 150 and I have 6 zombies. > > The Apache error_log shows some strange error: > > [ 2014-07-17 05:02:22.6186 28747/7f5909400720 agents/Watchdog/Main.cpp:631 > ]: All Phusion Passenger agents started! > [Thu Jul 17 05:02:22 2014] [notice] Digest: generating secret for digest > authentication ... > [Thu Jul 17 05:02:22 2014] [notice] Digest: done > [ 2014-07-17 05:02:22.6994 28769/7ffe47dac720 agents/Watchdog/Main.cpp:450 > ]: Options: { 'analytics_log_user' => 'nobody', 'default_group' => > 'nobody', 'default_python' => 'python', 'default_ruby' => 'ruby', > 'default_user' => 'nobody', 'log_level' => '0', 'max_instances_per_app' => > '0', 'max_pool_size' => '50', 'passenger_root' => > '/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini', > 'pool_idle_time' => '1500', 'temp_dir' => '/tmp', > 'union_station_gateway_address' => 'gateway.unionstationapp.com', > 'union_station_gateway_port' => '443', 'user_switching' => 'true', > 'web_server_pid' => '28767', 'web_server_type' => 'apache', > 'web_server_worker_gid' => '48', 'web_server_worker_uid' => '48' } > [ 2014-07-17 05:02:22.7038 28772/7f047e74b720 > agents/HelperAgent/Main.cpp:602 ]: PassengerHelperAgent online, listening > at unix:/tmp/passenger.1.0.28767/generation-0/request > [ 2014-07-17 05:02:22.7072 28778/7f8f0f2897e0 > agents/LoggingAgent/Main.cpp:318 ]: PassengerLoggingAgent online, listening > at unix:/tmp/passenger.1.0.28767/generation-0/logging > [ 2014-07-17 05:02:22.7075 28769/7ffe47dac720 agents/Watchdog/Main.cpp:631 > ]: All Phusion Passenger agents started! > [Thu Jul 17 05:02:22 2014] [notice] Apache/2.2.15 (Unix) DAV/2 > Phusion_Passenger/4.0.18 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- > resuming normal operations > [ 2014-07-17 05:02:24.2545 28772/7f047c852700 Pool2/Spawner.h:738 ]: [App > 28802 stdout] > [ 2014-07-17 05:02:25.1315 28772/7f047c852700 Pool2/SmartSpawner.h:301 ]: > Preloader for /etc/puppet/rack started on PID 28802, listening on > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.28802 > [ 2014-07-17 05:02:26.2069 28772/7f047c566700 Pool2/Spawner.h:738 ]: [App > 29081 stdout] > [ 2014-07-17 05:02:43.1300 28772/7f047c566700 Pool2/SmartSpawner.h:301 ]: > Preloader for /usr/share/foreman started on PID 29081, listening on > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.29094 > [ 2014-07-17 05:09:07.6952 28772/7f047c852700 Pool2/Spawner.h:738 ]: [App > 4599 stdout] > [ 2014-07-17 05:09:29.3475 28772/7f047c852700 Pool2/SmartSpawner.h:301 ]: > Preloader for /usr/share/foreman started on PID 4599, listening on > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.4632 > [ 2014-07-17 05:19:07.8990 28772/7f047c437700 Pool2/Spawner.h:738 ]: [App > 17278 stdout] > [ 2014-07-17 05:19:34.6716 28772/7f047c437700 Pool2/SmartSpawner.h:301 ]: > Preloader for /usr/share/foreman started on PID 17278, listening on > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.17306 > > Thanks, > > Royee > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas #lzap Zapletal

Royee_Tager · July 17, 2014, 3:04pm

[root@durham-foreman ~]# rpm -qa |grep -i foreman
foreman-vmware-1.5.0-1.el6.noarch
ruby193-rubygem-foreman_dhcp_browser-0.0.4-1.el6.noarch
rubygem-hammer_cli_foreman-0.1.0-1.el6.noarch
foreman-1.5.0-1.el6.noarch
foreman-proxy-1.5.0-1.el6.noarch
foreman-postgresql-1.5.0-1.el6.noarch
foreman-release-1.5.0-1.el6.noarch
foreman-compute-1.5.0-1.el6.noarch
foreman-installer-1.5.0-1.el6.noarch
foreman-console-1.5.0-1.el6.noarch
ruby193-rubygem-foreman_default_hostgroup-doc-2.0.1-1.el6.noarch
rubygem-foreman_api-0.1.11-1.el6.noarch
ruby193-rubygem-foreman_default_hostgroup-2.0.1-1.el6.noarch
[root@durham-foreman ~]# passenger --version
Phusion Passenger version 4.0.18

"Phusion Passenger" is a trademark of Hongli Lai & Ninh Bui.
[root@durham-foreman ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
[root@durham-foreman ~]#

בתאריך יום חמישי, 17 ביולי 2014 17:50:38 UTC+3, מאת Lukas Zapletal:

···

> > Hi, > > Foreman version? OS version? Passenger version? > > LZ > > On Wed, Jul 16, 2014 at 07:29:44PM -0700, Royee Tager wrote: > > Hello, > > > > Lately my Foreman server crashes. > > The top command shows that the load is 150 and I have 6 zombies. > > > > The Apache error_log shows some strange error: > > > > [ 2014-07-17 05:02:22.6186 28747/7f5909400720 > agents/Watchdog/Main.cpp:631 > > ]: All Phusion Passenger agents started! > > [Thu Jul 17 05:02:22 2014] [notice] Digest: generating secret for digest > > authentication ... > > [Thu Jul 17 05:02:22 2014] [notice] Digest: done > > [ 2014-07-17 05:02:22.6994 28769/7ffe47dac720 > agents/Watchdog/Main.cpp:450 > > ]: Options: { 'analytics_log_user' => 'nobody', 'default_group' => > > 'nobody', 'default_python' => 'python', 'default_ruby' => 'ruby', > > 'default_user' => 'nobody', 'log_level' => '0', 'max_instances_per_app' > => > > '0', 'max_pool_size' => '50', 'passenger_root' => > > > '/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini', > > > 'pool_idle_time' => '1500', 'temp_dir' => '/tmp', > > 'union_station_gateway_address' => 'gateway.unionstationapp.com', > > 'union_station_gateway_port' => '443', 'user_switching' => 'true', > > 'web_server_pid' => '28767', 'web_server_type' => 'apache', > > 'web_server_worker_gid' => '48', 'web_server_worker_uid' => '48' } > > [ 2014-07-17 05:02:22.7038 28772/7f047e74b720 > > agents/HelperAgent/Main.cpp:602 ]: PassengerHelperAgent online, > listening > > at unix:/tmp/passenger.1.0.28767/generation-0/request > > [ 2014-07-17 05:02:22.7072 28778/7f8f0f2897e0 > > agents/LoggingAgent/Main.cpp:318 ]: PassengerLoggingAgent online, > listening > > at unix:/tmp/passenger.1.0.28767/generation-0/logging > > [ 2014-07-17 05:02:22.7075 28769/7ffe47dac720 > agents/Watchdog/Main.cpp:631 > > ]: All Phusion Passenger agents started! > > [Thu Jul 17 05:02:22 2014] [notice] Apache/2.2.15 (Unix) DAV/2 > > Phusion_Passenger/4.0.18 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured > -- > > resuming normal operations > > [ 2014-07-17 05:02:24.2545 28772/7f047c852700 Pool2/Spawner.h:738 ]: > [App > > 28802 stdout] > > [ 2014-07-17 05:02:25.1315 28772/7f047c852700 Pool2/SmartSpawner.h:301 > ]: > > Preloader for /etc/puppet/rack started on PID 28802, listening on > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.28802 > > [ 2014-07-17 05:02:26.2069 28772/7f047c566700 Pool2/Spawner.h:738 ]: > [App > > 29081 stdout] > > [ 2014-07-17 05:02:43.1300 28772/7f047c566700 Pool2/SmartSpawner.h:301 > ]: > > Preloader for /usr/share/foreman started on PID 29081, listening on > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.29094 > > [ 2014-07-17 05:09:07.6952 28772/7f047c852700 Pool2/Spawner.h:738 ]: > [App > > 4599 stdout] > > [ 2014-07-17 05:09:29.3475 28772/7f047c852700 Pool2/SmartSpawner.h:301 > ]: > > Preloader for /usr/share/foreman started on PID 4599, listening on > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.4632 > > [ 2014-07-17 05:19:07.8990 28772/7f047c437700 Pool2/Spawner.h:738 ]: > [App > > 17278 stdout] > > [ 2014-07-17 05:19:34.6716 28772/7f047c437700 Pool2/SmartSpawner.h:301 > ]: > > Preloader for /usr/share/foreman started on PID 17278, listening on > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.17306 > > > > Thanks, > > > > Royee > > > > -- > > You received this message because you are subscribed to the Google > Groups "Foreman users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-user...@googlegroups.com . > > To post to this group, send email to forema...@googlegroups.com > . > > Visit this group at http://groups.google.com/group/foreman-users. > > For more options, visit https://groups.google.com/d/optout. > > -- > Later, > Lukas #lzap Zapletal >

lzap · July 17, 2014, 3:09pm

Have you changed passenger or httpd configuration? What are your
passenger limits? Maximum amount of processes, threadholds and that
kind of stuff?

Also is SELinux in enforcing? Any denials?

LZ

···

On Thu, Jul 17, 2014 at 08:04:15AM -0700, Royee Tager wrote: > [root@durham-foreman ~]# rpm -qa |grep -i foreman > foreman-vmware-1.5.0-1.el6.noarch > ruby193-rubygem-foreman_dhcp_browser-0.0.4-1.el6.noarch > rubygem-hammer_cli_foreman-0.1.0-1.el6.noarch > foreman-1.5.0-1.el6.noarch > foreman-proxy-1.5.0-1.el6.noarch > foreman-postgresql-1.5.0-1.el6.noarch > foreman-release-1.5.0-1.el6.noarch > foreman-compute-1.5.0-1.el6.noarch > foreman-installer-1.5.0-1.el6.noarch > foreman-console-1.5.0-1.el6.noarch > ruby193-rubygem-foreman_default_hostgroup-doc-2.0.1-1.el6.noarch > rubygem-foreman_api-0.1.11-1.el6.noarch > ruby193-rubygem-foreman_default_hostgroup-2.0.1-1.el6.noarch > [root@durham-foreman ~]# passenger --version > Phusion Passenger version 4.0.18 > > "Phusion Passenger" is a trademark of Hongli Lai & Ninh Bui. > [root@durham-foreman ~]# cat /etc/redhat-release > CentOS release 6.5 (Final) > [root@durham-foreman ~]# > > בתאריך יום חמישי, 17 ביולי 2014 17:50:38 UTC+3, מאת Lukas Zapletal: > > > > Hi, > > > > Foreman version? OS version? Passenger version? > > > > LZ > > > > On Wed, Jul 16, 2014 at 07:29:44PM -0700, Royee Tager wrote: > > > Hello, > > > > > > Lately my Foreman server crashes. > > > The top command shows that the load is 150 and I have 6 zombies. > > > > > > The Apache error_log shows some strange error: > > > > > > [ 2014-07-17 05:02:22.6186 28747/7f5909400720 > > agents/Watchdog/Main.cpp:631 > > > ]: All Phusion Passenger agents started! > > > [Thu Jul 17 05:02:22 2014] [notice] Digest: generating secret for digest > > > authentication ... > > > [Thu Jul 17 05:02:22 2014] [notice] Digest: done > > > [ 2014-07-17 05:02:22.6994 28769/7ffe47dac720 > > agents/Watchdog/Main.cpp:450 > > > ]: Options: { 'analytics_log_user' => 'nobody', 'default_group' => > > > 'nobody', 'default_python' => 'python', 'default_ruby' => 'ruby', > > > 'default_user' => 'nobody', 'log_level' => '0', 'max_instances_per_app' > > => > > > '0', 'max_pool_size' => '50', 'passenger_root' => > > > > > '/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini', > > > > > 'pool_idle_time' => '1500', 'temp_dir' => '/tmp', > > > 'union_station_gateway_address' => 'gateway.unionstationapp.com', > > > 'union_station_gateway_port' => '443', 'user_switching' => 'true', > > > 'web_server_pid' => '28767', 'web_server_type' => 'apache', > > > 'web_server_worker_gid' => '48', 'web_server_worker_uid' => '48' } > > > [ 2014-07-17 05:02:22.7038 28772/7f047e74b720 > > > agents/HelperAgent/Main.cpp:602 ]: PassengerHelperAgent online, > > listening > > > at unix:/tmp/passenger.1.0.28767/generation-0/request > > > [ 2014-07-17 05:02:22.7072 28778/7f8f0f2897e0 > > > agents/LoggingAgent/Main.cpp:318 ]: PassengerLoggingAgent online, > > listening > > > at unix:/tmp/passenger.1.0.28767/generation-0/logging > > > [ 2014-07-17 05:02:22.7075 28769/7ffe47dac720 > > agents/Watchdog/Main.cpp:631 > > > ]: All Phusion Passenger agents started! > > > [Thu Jul 17 05:02:22 2014] [notice] Apache/2.2.15 (Unix) DAV/2 > > > Phusion_Passenger/4.0.18 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured > > -- > > > resuming normal operations > > > [ 2014-07-17 05:02:24.2545 28772/7f047c852700 Pool2/Spawner.h:738 ]: > > [App > > > 28802 stdout] > > > [ 2014-07-17 05:02:25.1315 28772/7f047c852700 Pool2/SmartSpawner.h:301 > > ]: > > > Preloader for /etc/puppet/rack started on PID 28802, listening on > > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.28802 > > > [ 2014-07-17 05:02:26.2069 28772/7f047c566700 Pool2/Spawner.h:738 ]: > > [App > > > 29081 stdout] > > > [ 2014-07-17 05:02:43.1300 28772/7f047c566700 Pool2/SmartSpawner.h:301 > > ]: > > > Preloader for /usr/share/foreman started on PID 29081, listening on > > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.29094 > > > [ 2014-07-17 05:09:07.6952 28772/7f047c852700 Pool2/Spawner.h:738 ]: > > [App > > > 4599 stdout] > > > [ 2014-07-17 05:09:29.3475 28772/7f047c852700 Pool2/SmartSpawner.h:301 > > ]: > > > Preloader for /usr/share/foreman started on PID 4599, listening on > > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.4632 > > > [ 2014-07-17 05:19:07.8990 28772/7f047c437700 Pool2/Spawner.h:738 ]: > > [App > > > 17278 stdout] > > > [ 2014-07-17 05:19:34.6716 28772/7f047c437700 Pool2/SmartSpawner.h:301 > > ]: > > > Preloader for /usr/share/foreman started on PID 17278, listening on > > > unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.17306 > > > > > > Thanks, > > > > > > Royee > > > > > > -- > > > You received this message because you are subscribed to the Google > > Groups "Foreman users" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > an email to foreman-user...@googlegroups.com . > > > To post to this group, send email to forema...@googlegroups.com > > . > > > Visit this group at http://groups.google.com/group/foreman-users. > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > Later, > > Lukas #lzap Zapletal > > > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas #lzap Zapletal

Royee_Tager · July 17, 2014, 3:16pm

Hi Lukas,

Since currently my Foreman server manages 1921, I had to change the values
of the passanger and the httpd.
Below are the values

http.conf:

···

=========== StartServers 8 MinSpareServers 5 MaxSpareServers 20 ###ServerLimit 256 ServerLimit 128 #ServerLimit 80000 ###MaxClients 256 MaxClients 128 #MaxClients 80000 ###MaxRequestsPerChild 4000 MaxRequestsPerChild 2000

worker MPM

StartServers: initial number of server processes to start

MaxClients: maximum number of simultaneous client connections

MinSpareThreads: minimum number of worker threads which are kept spare

MaxSpareThreads: maximum number of worker threads which are kept spare

ThreadsPerChild: constant number of worker threads in each server process

MaxRequestsPerChild: maximum number of requests a server process serves

StartServers 4 ###MaxClients 300 MaxClients 400 #MaxClients 80000 MinSpareThreads 25 MaxSpareThreads 75 ###ThreadsPerChild 25 ThreadsPerChild 15 MaxRequestsPerChild 0

puppet.conf:

you probably want to tune these settings

PassengerHighPerformance on
#PassengerMaxPoolSize 12
#PassengerMaxPoolSize 500
PassengerMaxPoolSize 50
PassengerPoolIdleTime 1500
PassengerMaxRequests 500
PassengerStatThrottleRate 120

[root@durham-foreman ~]# getenforce
Disabled
[root@durham-foreman ~]#

בתאריך יום חמישי, 17 ביולי 2014 18:09:14 UTC+3, מאת Lukas Zapletal:

Have you changed passenger or httpd configuration? What are your
passenger limits? Maximum amount of processes, threadholds and that
kind of stuff?

Also is SELinux in enforcing? Any denials?

LZ

On Thu, Jul 17, 2014 at 08:04:15AM -0700, Royee Tager wrote:

[root@durham-foreman ~]# rpm -qa |grep -i foreman
foreman-vmware-1.5.0-1.el6.noarch
ruby193-rubygem-foreman_dhcp_browser-0.0.4-1.el6.noarch
rubygem-hammer_cli_foreman-0.1.0-1.el6.noarch
foreman-1.5.0-1.el6.noarch
foreman-proxy-1.5.0-1.el6.noarch
foreman-postgresql-1.5.0-1.el6.noarch
foreman-release-1.5.0-1.el6.noarch
foreman-compute-1.5.0-1.el6.noarch
foreman-installer-1.5.0-1.el6.noarch
foreman-console-1.5.0-1.el6.noarch
ruby193-rubygem-foreman_default_hostgroup-doc-2.0.1-1.el6.noarch
rubygem-foreman_api-0.1.11-1.el6.noarch
ruby193-rubygem-foreman_default_hostgroup-2.0.1-1.el6.noarch
[root@durham-foreman ~]# passenger --version
Phusion Passenger version 4.0.18

“Phusion Passenger” is a trademark of Hongli Lai & Ninh Bui.
[root@durham-foreman ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
[root@durham-foreman ~]#

בתאריך יום חמישי, 17 ביולי 2014 17:50:38 UTC+3, מאת Lukas Zapletal:

Hi,

Foreman version? OS version? Passenger version?

LZ

On Wed, Jul 16, 2014 at 07:29:44PM -0700, Royee Tager wrote:

Hello,

Lately my Foreman server crashes.
The top command shows that the load is 150 and I have 6 zombies.

The Apache error_log shows some strange error:

[ 2014-07-17 05:02:22.6186 28747/7f5909400720
agents/Watchdog/Main.cpp:631
]: All Phusion Passenger agents started!
[Thu Jul 17 05:02:22 2014] [notice] Digest: generating secret for
digest

authentication …
[Thu Jul 17 05:02:22 2014] [notice] Digest: done
[ 2014-07-17 05:02:22.6994 28769/7ffe47dac720
agents/Watchdog/Main.cpp:450
]: Options: { ‘analytics_log_user’ => ‘nobody’, ‘default_group’ =>
‘nobody’, ‘default_python’ => ‘python’, ‘default_ruby’ => ‘ruby’,
‘default_user’ => ‘nobody’, ‘log_level’ => ‘0’,
‘max_instances_per_app’

=>

‘0’, ‘max_pool_size’ => ‘50’, ‘passenger_root’ =>

‘/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini’,

‘pool_idle_time’ => ‘1500’, ‘temp_dir’ => ‘/tmp’,
‘union_station_gateway_address’ => ‘gateway.unionstationapp.com’,
‘union_station_gateway_port’ => ‘443’, ‘user_switching’ => ‘true’,
‘web_server_pid’ => ‘28767’, ‘web_server_type’ => ‘apache’,
‘web_server_worker_gid’ => ‘48’, ‘web_server_worker_uid’ => ‘48’ }
[ 2014-07-17 05:02:22.7038 28772/7f047e74b720
agents/HelperAgent/Main.cpp:602 ]: PassengerHelperAgent online,
listening
at unix:/tmp/passenger.1.0.28767/generation-0/request
[ 2014-07-17 05:02:22.7072 28778/7f8f0f2897e0
agents/LoggingAgent/Main.cpp:318 ]: PassengerLoggingAgent online,
listening
at unix:/tmp/passenger.1.0.28767/generation-0/logging
[ 2014-07-17 05:02:22.7075 28769/7ffe47dac720
agents/Watchdog/Main.cpp:631
]: All Phusion Passenger agents started!
[Thu Jul 17 05:02:22 2014] [notice] Apache/2.2.15 (Unix) DAV/2
Phusion_Passenger/4.0.18 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips
configured

–

resuming normal operations
[ 2014-07-17 05:02:24.2545 28772/7f047c852700 Pool2/Spawner.h:738 ]:
[App
28802 stdout]
[ 2014-07-17 05:02:25.1315 28772/7f047c852700
Pool2/SmartSpawner.h:301

]:

Preloader for /etc/puppet/rack started on PID 28802, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.28802
[ 2014-07-17 05:02:26.2069 28772/7f047c566700 Pool2/Spawner.h:738 ]:
[App
29081 stdout]
[ 2014-07-17 05:02:43.1300 28772/7f047c566700
Pool2/SmartSpawner.h:301

]:

Preloader for /usr/share/foreman started on PID 29081, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.29094
[ 2014-07-17 05:09:07.6952 28772/7f047c852700 Pool2/Spawner.h:738 ]:
[App
4599 stdout]
[ 2014-07-17 05:09:29.3475 28772/7f047c852700
Pool2/SmartSpawner.h:301

]:

Preloader for /usr/share/foreman started on PID 4599, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.4632
[ 2014-07-17 05:19:07.8990 28772/7f047c437700 Pool2/Spawner.h:738 ]:
[App
17278 stdout]
[ 2014-07-17 05:19:34.6716 28772/7f047c437700
Pool2/SmartSpawner.h:301

]:

Preloader for /usr/share/foreman started on PID 17278, listening on
unix:/tmp/passenger.1.0.28767/generation-0/backends/preloader.17306

Thanks,

Royee

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it,
send

an email to foreman-user...@googlegroups.com <javascript:>.

To post to this group, send email to forema...@googlegroups.com
<javascript:>.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-user...@googlegroups.com <javascript:>.
To post to this group, send email to forema...@googlegroups.com
<javascript:>.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas #lzap Zapletal

lzap · July 17, 2014, 3:43pm

> # you probably want to tune these settings
> PassengerHighPerformance on
> #PassengerMaxPoolSize 12
> #PassengerMaxPoolSize 500
> PassengerMaxPoolSize 50
> PassengerPoolIdleTime 1500
> PassengerMaxRequests 500
> PassengerStatThrottleRate 120

What is the hardware you run on? Memory?

Passenger uses copy on write memory to save some when new process is
forked but thank to the GC implementation in Ruby 1.9.3 it eventually
consumes whole process block (no memory shared) which is for Foreman
about 200MB (very rough estimate from my current development setups).

That means if you set 50 simultaneous processes, that can eventually
(not from the very beginning) require 10 GB of memory assuming that
Foreman process can take as much as 200MB (which is my wild guess - I'd
appreciate some more exact measurements from other production
instances).

Therefore I'd recommend to revisit this setting according to what your
process instances require.

Also there are some native optimalizations in the Passenger. I've seen
some versions of Passenger packaged by us without these native libraries
on proper paths and Ruby was not loading them. In that case, Passenger
issues some working in the httpd logs and carries on in the non-native
(slower) mode. Check for that too.

Is my assumption correct that the load is done by puppet master and not
the foreman application?

···

-- Later, Lukas #lzap Zapletal

Royee_Tager · July 18, 2014, 11:16am

Hi Lukas,

My Foreman / Puppet server has crashed again. It crashes everyday.
My server runs on VM (VMware). The VM has 64GB RAM and 16 CPUs.

Since I'm not familiar with Passanger I didn't understand your previous
answer and I don't know how to make the appropriate optimization.

Could you please assist?

בתאריך יום חמישי, 17 ביולי 2014 18:43:34 UTC+3, מאת Lukas Zapletal:

···

> > > # you probably want to tune these settings > > PassengerHighPerformance on > > #PassengerMaxPoolSize 12 > > #PassengerMaxPoolSize 500 > > PassengerMaxPoolSize 50 > > PassengerPoolIdleTime 1500 > > PassengerMaxRequests 500 > > PassengerStatThrottleRate 120 > > What is the hardware you run on? Memory? > > Passenger uses copy on write memory to save some when new process is > forked but thank to the GC implementation in Ruby 1.9.3 it eventually > consumes whole process block (no memory shared) which is for Foreman > about 200MB (very rough estimate from my current development setups). > > That means if you set 50 simultaneous processes, that can *eventually* > (not from the very beginning) require 10 GB of memory assuming that > Foreman process can take as much as 200MB (which is my wild guess - I'd > appreciate some more exact measurements from other production > instances). > > Therefore I'd recommend to revisit this setting according to what your > process instances require. > > Also there are some native optimalizations in the Passenger. I've seen > some versions of Passenger packaged by us without these native libraries > on proper paths and Ruby was not loading them. In that case, Passenger > issues some working in the httpd logs and carries on in the non-native > (slower) mode. Check for that too. > > Is my assumption correct that the load is done by puppet master and not > the foreman application? > > -- > Later, > Lukas #lzap Zapletal >

lzap · July 21, 2014, 9:44am

Hey,

> Since I'm not familiar with Passanger I didn't understand your previous
> answer and I don't know how to make the appropriate optimization.

What is your memory consumption per Foreman passenger process?

> Could you please assist?

What I think might help would be trying to lower this number to the
point that Foreman is stable and responsive enough:

> > > #PassengerMaxPoolSize 12
> > > PassengerMaxPoolSize 50

I am no Passenger expert as well, maybe other production admins can chip
in here.

···

-- Later, Lukas #lzap Zapletal

Royee_Tager · July 28, 2014, 5:12am

Hi Lukas,

Thanks for your assistance.
After a long investiation I'm glad to update you that the problem was not
related to the passanger at all.
As I wrote last week, my Puppet & Foreman are installed on a VM (VMware).
We use VDP (Vsphere Data Protection ) for backing up our VMs.
For some reason every time when the backup started (snapshot) the CPU load
reached 150.
I solved the problem by removing the the VM from the backup policy.
Ever since (8 days ago) the server has not crashed.

Thanks,

Royee Tager

בתאריך יום שני, 21 ביולי 2014 12:44:36 UTC+3, מאת Lukas Zapletal:

···

> > Hey, > > > Since I'm not familiar with Passanger I didn't understand your previous > > answer and I don't know how to make the appropriate optimization. > > What is your memory consumption per Foreman passenger process? > > > Could you please assist? > > What *I* think might help would be trying to lower this number to the > point that Foreman is stable and responsive enough: > > > > > #PassengerMaxPoolSize 12 > > > > PassengerMaxPoolSize 50 > > I am no Passenger expert as well, maybe other production admins can chip > in here. > > -- > Later, > Lukas #lzap Zapletal >

lzap · July 28, 2014, 11:31am

Out of my curiosity, can you share some numbers with us?

How many passenger instances are running?

Out of these, which are Foreman and Puppet master?

How much memory each instance type consume in average?

Since your limit is 50 instances, how many does it spawn during peaks?

Glad you sorted it out.

LZ

···

On Sun, Jul 27, 2014 at 10:12:26PM -0700, Royee Tager wrote: > Hi Lukas, > > Thanks for your assistance. > After a long investiation I'm glad to update you that the problem was not > related to the passanger at all. > As I wrote last week, my Puppet & Foreman are installed on a VM (VMware). > We use VDP (Vsphere Data Protection ) for backing up our VMs. > For some reason every time when the backup started (snapshot) the CPU load > reached 150. > I solved the problem by removing the the VM from the backup policy. > Ever since (8 days ago) the server has not crashed. > > Thanks, > > Royee Tager > > בתאריך יום שני, 21 ביולי 2014 12:44:36 UTC+3, מאת Lukas Zapletal: > > > > Hey, > > > > > Since I'm not familiar with Passanger I didn't understand your previous > > > answer and I don't know how to make the appropriate optimization. > > > > What is your memory consumption per Foreman passenger process? > > > > > Could you please assist? > > > > What *I* think might help would be trying to lower this number to the > > point that Foreman is stable and responsive enough: > > > > > > > #PassengerMaxPoolSize 12 > > > > > PassengerMaxPoolSize 50 > > > > I am no Passenger expert as well, maybe other production admins can chip > > in here. > > > > -- > > Later, > > Lukas #lzap Zapletal > > > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas #lzap Zapletal

Victor_Vedmich · March 18, 2015, 11:47am

Hello guys, I have the same issue, but in my case it's: clean VM with
Centos 6.5 2CPU and 4Gb RAM, just installed puppet and foreman, I log in on
foreman dashboard but after few min puppet master and dashboard were
broken. And in apache log I see the following stacktrace, and the
stacktrace repeat after every restart apache.

[ 2015-03-18 07:25:17.3464 1275/7f7140a9e720 agents/Watchdog/Main.cpp:450 ]:
Options: { 'analytics_log_user' => 'nobody', 'default_group' => 'nobody',
'default_python' => 'python', 'default_ruby' => 'ruby', 'default_user' =>
'nobody', 'log_level' => '0', 'max_instances_per_app' => '0',
'max_pool_size' => '6', 'passenger_root' =>
'/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini'
, 'pool_idle_time' => '300', 'prestart_urls' =>
'aHR0cDovL3B1cHBldDAxNy5ub3ZhbG9jYWwAaHR0cHM6Ly9wdXBwZXQwMTcubm92YWxvY2FsAA=='
, 'temp_dir' => '/tmp', 'union_station_gateway_address' =>
'gateway.unionstationapp.com', 'union_station_gateway_port' => '443',
'user_switching' => 'true', 'web_server_pid' => '1274', 'web_server_type' =>
'apache', 'web_server_worker_gid' => '48', 'web_server_worker_uid' => '48' }
[ 2015-03-18 07:25:18.2326 1278/7f1f98666720 agents/HelperAgent/Main.cpp:602
]: PassengerHelperAgent online, listening at unix:/tmp/passenger.1.0.1274/
generation-0/request
[ 2015-03-18 07:25:18.3844 1283/7f6286b697e0 agents/LoggingAgent/Main.cpp:
318 ]: PassengerLoggingAgent online, listening at unix:/tmp/passenger.1.0.
1274/generation-0/logging
[ 2015-03-18 07:25:18.3847 1275/7f7140a9e720 agents/Watchdog/Main.cpp:631 ]:
All Phusion Passenger agents started!
[Wed Mar 18 07:25:18 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/
sbin/suexec)
[Wed Mar 18 07:25:18 2015] [warn] module passenger_module is already loaded,
skipping
[Wed Mar 18 07:25:18 2015] [notice] Digest: generating secret for digest
authentication …
[Wed Mar 18 07:25:18 2015] [notice] Digest: done
[ 2015-03-18 07:25:18.5437 1300/7f37ef8f3720 agents/Watchdog/Main.cpp:450 ]:
Options: { 'analytics_log_user' => 'nobody', 'default_group' => 'nobody',
'default_python' => 'python', 'default_ruby' => 'ruby', 'default_user' =>
'nobody', 'log_level' => '0', 'max_instances_per_app' => '0',
'max_pool_size' => '6', 'passenger_root' =>
'/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini'
, 'pool_idle_time' => '300', 'prestart_urls' =>
'aHR0cDovL3B1cHBldDAxNy5ub3ZhbG9jYWwAaHR0cHM6Ly9wdXBwZXQwMTcubm92YWxvY2FsAA=='
, 'temp_dir' => '/tmp', 'union_station_gateway_address' =>
'gateway.unionstationapp.com', 'union_station_gateway_port' => '443',
'user_switching' => 'true', 'web_server_pid' => '1295', 'web_server_type' =>
'apache', 'web_server_worker_gid' => '48', 'web_server_worker_uid' => '48' }
[ 2015-03-18 07:25:18.5678 1304/7fda94625720 agents/HelperAgent/Main.cpp:602
]: PassengerHelperAgent online, listening at unix:/tmp/passenger.1.0.1295/
generation-0/request
[ 2015-03-18 07:25:18.5746 1310/7f75c23ba7e0 agents/LoggingAgent/Main.cpp:
318 ]: PassengerLoggingAgent online, listening at unix:/tmp/passenger.1.0.
1295/generation-0/logging
[ 2015-03-18 07:25:18.5748 1300/7f37ef8f3720 agents/Watchdog/Main.cpp:631 ]:
All Phusion Passenger agents started!
[Wed Mar 18 07:25:18 2015] [notice] Apache/2.2.15 (Unix) DAV/2
Phusion_Passenger/4.0.18 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_wsgi/3.4
Python/2.6.6 configured – resuming normal operations
[ 2015-03-18 07:25:20.3115 1304/7fda9452e700 Pool2/Spawner.h:738 ]: [App
1354 stdout]
[ 2015-03-18 07:25:48.5184 1294/7f7140a9e720 agents/Watchdog/Main.cpp:337 ]:
Some Phusion Passenger agent processes did not exit in time, forcefully
shutting down all.
[ 2015-03-18 07:26:01.7726 1304/7fda9452e700 Pool2/SmartSpawner.h:301 ]:
Preloader for /usr/share/foreman started on PID 1354, listening on unix:
/tmp/passenger.1.0.1295/generation-0/backends/preloader.1370
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/prespawn:105:in initialize': Connection refused - connect(2) (Errno::ECONNREFUSED) from /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/prespawn:105:innew'
from
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/prespawn:105:in
connect' from /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/ prespawn:112:inconnect'
from
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/prespawn:86:in socket' from /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/prespawn:90:inhead_request'
from /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/
prespawn:145