Hello,
i'm experiencing performance issues on my Katello 2.3 setup on RHEL 6. My
server is a 4cpu 19gb of ram virtual machine. The VM performs well, no
particular issues.
The problem i'm experiencing is a huge queuing on passenger-status.
With the default configuration i expecienced tons of:
"Error 503 on SERVER: This website is under heavy load"
I made some search and after tuning resources (added more ram to get 19gb),
i changed Passenger configuration.
Now my passenger config is this:
Global:
PassengerMaxRequests 1000
PassengerMaxPoolSize 20
PassengerMaxRequestQueueSize 0
On foreman vhost:
PassengerMinInstances 4
PassengerStartTimeout 600
Is not infrequent that I find in the output of passenger-status "Requests
in queue" value related to puppet master for more than 5 minutes greater
than 200. All workers seems to be busy (Sessions: 1).
Checking why the workers are busy appears that are all waiting for the
output of node.rb execution (did a check using strace).
This issue leads to several hosts going "out of sync" and the foreman web
interface doesn't load.
Anyone is experiencing similar issues? Any suggestions for tuning? I'm
going too big (registered hosts will be around 1500 when setup is
completed) for a single instance?
Thank you
> Hello,
>
> i'm experiencing performance issues on my Katello 2.3 setup on RHEL 6. My
> server is a 4cpu 19gb of ram virtual machine. The VM performs well, no
> particular issues.
>
> The problem i'm experiencing is a huge queuing on passenger-status.
>
> With the default configuration i expecienced tons of:
>
> "Error 503 on SERVER: This website is under heavy load"
>
> I made some search and after tuning resources (added more ram to get 19gb),
> i changed Passenger configuration.
>
> Now my passenger config is this:
>
> Global:
>
> PassengerMaxRequests 1000
> PassengerMaxPoolSize 20
> PassengerMaxRequestQueueSize 0
>
> On foreman vhost:
> PassengerMinInstances 4
> PassengerStartTimeout 600
>
>
> Is not infrequent that I find in the output of passenger-status "Requests
> in queue" value related to puppet master for more than 5 minutes greater
> than 200. All workers seems to be busy (Sessions: 1).
We have a redmine issue to be able to configure PassengerMaxPoolSize,
Bug #14023: puppet-foreman does not allow for configuration of PassengerMaxPoolSize - Installer - Foreman.
I think the PassengerMaxRequests is only needed if you suspect a memory
leak, since it will close and re-open a new passenger process after X
requests. It probably doesn't hurt to have it in there though,
especially since you have 20 workers.
I am not sure about the MaxRequestQueueSize, it may be needed but it
sounds like it's more of a workaround for the node.rb issue than
something you'd want to keep there permanently.
>
> Checking why the workers are busy appears that are all waiting for the
> output of node.rb execution (did a check using strace).
I'm not an expert in this area by any means, but I'm curious how many
environments you have. What is the output of "ls
/etc/puppet/environments/ | wc -l" ?
···
On 03/21/2016 08:05 AM, Luca Lorenzetto wrote:
This issue leads to several hosts going “out of sync” and the foreman web
interface doesn’t load.
Anyone is experiencing similar issues? Any suggestions for tuning? I’m
going too big (registered hosts will be around 1500 when setup is
completed) for a single instance?
Thank you
[cut]
>
>
>
> We have a redmine issue to be able to configure PassengerMaxPoolSize,
> Bug #14023: puppet-foreman does not allow for configuration of PassengerMaxPoolSize - Installer - Foreman.
>
>
>
I'll take a look. Thank you.
>
> >
> > Checking why the workers are busy appears that are all waiting for the
> > output of node.rb execution (did a check using strace).
>
> I'm not an expert in this area by any means, but I'm curious how many
> environments you have. What is the output of "ls
> /etc/puppet/environments/ | wc -l" ?
>
>
[root@katelloserver ~]# ls -l /etc/puppet/environments/
total 36
drwxr-xr-x 3 apache apache 4096 Mar 17 17:43 KT_org_Library_Puppet_12
drwxr-xr-x 3 apache apache 4096 Mar 17 17:44 KT_org_Produzione_Puppet_12
drwxr-xr-x 3 apache apache 4096 Mar 17 17:43 KT_org_Sviluppo_Puppet_12
drwxr-xr-x 3 apache apache 4096 Mar 17 17:44 KT_org_Test_Puppet_12
drwx------ 2 root root 16384 Dec 22 08:58 lost+found
drwxr-xr-x 3 root root 4096 Oct 16 16:22 production
These are my environments.
production is not used, it's here because we allow hosts to register as
default and then with foreman we set the correct environment.
Luca
···
Il giorno lunedì 21 marzo 2016 15:51:26 UTC+1, Chris Duryee ha scritto:
Another thing i noticed:
I'm updating around 50 hosts in parallel. If puppet-master uses a lot of
passenger slots, there's no way to download rpm files and upgrade fails. I
applied another fix:
PassengerMaxInstancesPerApp 16
in 25-puppet.conf to limit the usage of slots by puppet. I hope this will
limit its eager behaviour.
