Foreman httpd Performance on large environment - "heavy load" message

Support
1

Problem:
Foreman not available and message “heavy load” on web access

Expected outcome:
As soon as possible, this issue in production environment

Foreman and Proxy versions:
V1.19.0

Foreman and Proxy plugin versions:
Katello V3.8.0
Foreman remote execution 1.5.6
Forman task 0.13.4
Bastion 6.1.1

Distribution and version:
CentOS V7.5

Other relevant data:

2

Adding some information about issue

##Problem description
Time to time foreman httpd server hang with message “heavy load” or is not possible to join from API or Browse request.

I have activated http module mod_status and display worker
Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Phusion_Passenger/4.0.53

Server MPM: prefork

Server Built: Jun 27 2018 13:48:59

Current Time: Monday, 02-Dec-2019 10:59:51 CET

Restart Time: Monday, 02-Dec-2019 09:44:13 CET

Parent Server Config. Generation: 1

Parent Server MPM Generation: 0

Server uptime: 1 hour 15 minutes 37 seconds

Server load: 1.88 1.24 1.22

Total accesses: 73550 - Total Traffic: 2.5 GB

CPU Usage: u220.33 s20.31 cu0 cs0 - 5.3% CPU load

16.2 requests/sec - 0.6 MB/second - 35.1 kB/request

19 requests currently being processed, 14 idle workers

__.WKR_RW_WRWRRKWRW.K__W_____W_.RW_._........................... ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................

##Investigation

  • Time to time worker number increaze drasticaly and when max is targeted httpd seem to hang.
  • to resolve temporary I restart httpd server

Action already done

I have increaze passenger worker form 256 to 512 but the pissue still present.

Help requested

  • Do you have idea and solution for tunning foreman, passenger, httpd in large environement (more than 3500 client).

I need your help to solve this issue in production environement

3

This is what we use:

$passenger_instances = ceiling($::facts['processorcount'] * 1.5)

passenger_min_instances => $passenger_instances,

PassengerMaxPoolSize ${passenger_instances}
PassengerMaxRequestQueueSize 20

Do you know the passenger-status command? That helps to debug these issues.

4

Hello, in first thanks for your help.

I have increaze passenger nuber as sugested. But when I try to use:
$> passenger-status
ERROR: Phusion Passenger doesn’t seem to be running.

5

That’s because httpd runs in a private temp via systemd, you can disable that if you want:

  systemd::unit_file { 'passenger-disable-private-temp.service':
    path    => '/etc/systemd/system/httpd.service.d',
    content => "[Service]\nPrivateTmp=false",
  }