Performance tweaks?

Support
1

Hi,

I've recently suffered from a really annoying problem.

"new nodes did not register properly with puppet" or at least that was
my description.
So old nodes worked good, new nodes had no catalog. After a lot of
digging I found out that the ENC was not working and problem was it got
timed out against the URL(maybe there should be a syslog error when that
happens?) and it started using the cached yaml files instead.
So this issue appeared after I did a yum upgrade on my CentOS 6. Somehow
the updates caused foreman to respond slower than before. So the timeout
that was set to 3 seconds always hit and I was force to set it to 10
seconds. After that it worked.
I've also buffed my virtual server from 1 vCPU to 2 vCPU and from 1GB
RAM to 4GB RAM. Still curl against foreman takes 5 seconds to get the
yaml output.

So after this horrific story… I actually have a question. I run puppet
master + apache + passenger + foreman + mysql on the same server. Does
any one have a performance tweak guide? Before I have only followed the
passenger tweaks that are on puppetlabs wiki…

Regards,
/Raboo

2

A few things come to mind.

How many nodes in puppet do you have?
Is foreman configured to use sqlite? If so switch to use mysql.
2vCPU and 4GB ram won't do you any good if the underlying disk IO is really crappy. Is your time on the puppet master always drifting?
Are you using storeconfigs? If so I have heard this can suck up a lot of resources if you have enough nodes.

I would separate your puppet master and foreman to different machines. It does require more work but you will be able to separate the workload around on your vm hosts easier down the road.

I don't remember how long it takes to get the yaml file but I do remember it was quick. Are you able to show us your YAML file?

Also are you now running the equivalent of CentOS 6.2? I wonder what would have changed that would cause the slowdown.

Try the following to rule out a apache/passenger issues.

stop the foreman instance in apache and run foreman under webrick server

cd into your foreman directory.

RAILS_ENV=production rails s (for foreman 0.5 – development)
script/server -e production -p 3000 -d (foreman version < 0.5)

rerun your enc yaml script and see if that improves anything. If this is the case something could be wrong with apache/passenger.

Corey Osman
corey@logicminds.biz

Green IT and Datacenter Automation Specialist

··· On May 10, 2012, at 3:16 PM, Elias Abacioglu wrote:

Hi,

I’ve recently suffered from a really annoying problem.

“new nodes did not register properly with puppet” or at least that was my description.
So old nodes worked good, new nodes had no catalog. After a lot of digging I found out that the ENC was not working and problem was it got timed out against the URL(maybe there should be a syslog error when that happens?) and it started using the cached yaml files instead.
So this issue appeared after I did a yum upgrade on my CentOS 6. Somehow the updates caused foreman to respond slower than before. So the timeout that was set to 3 seconds always hit and I was force to set it to 10 seconds. After that it worked.
I’ve also buffed my virtual server from 1 vCPU to 2 vCPU and from 1GB RAM to 4GB RAM. Still curl against foreman takes 5 seconds to get the yaml output.

So after this horrific story… I actually have a question. I run puppet master + apache + passenger + foreman + mysql on the same server. Does any one have a performance tweak guide? Before I have only followed the passenger tweaks that are on puppetlabs wiki…

Regards,
/Raboo

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

3

ב-11 במאי 2012, בשעה 01:16, Elias Abacioglu <elias.rabi@gmail.com> כתב/ה:

> Hi,
>
> I've recently suffered from a really annoying problem.
>
> "new nodes did not register properly with puppet" or at least that was my description.
> So old nodes worked good, new nodes had no catalog. After a lot of digging I found out that the ENC was not working and problem was it got timed out against the URL(maybe there should be a syslog error when that happens?) and it started using the cached yaml files instead.
> So this issue appeared after I did a yum upgrade on my CentOS 6. Somehow the updates caused foreman to respond slower than before. So the timeout that was set to 3 seconds always hit and I was force to set it to 10 seconds. After that it worked.
> I've also buffed my virtual server from 1 vCPU to 2 vCPU and from 1GB RAM to 4GB RAM. Still curl against foreman takes 5 seconds to get the yaml output.
>
> So after this horrific story… I actually have a question. I run puppet master + apache + passenger + foreman + mysql on the same server. Does any one have a performance tweak guide? Before I have only followed the passenger tweaks that are on puppetlabs wiki…
>
It's hard to say why (from the description) but 5 seconds is not acceptable.
Can you describe your setup? Is your machine swapping a lot? If you could go over the logs and see what takes the longest that might be a starting point.

Ohad

··· > Regards, > /Raboo > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To post to this group, send email to foreman-users@googlegroups.com. > To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en. >
4

Hi,

After more digging I was able to get to the root cause of the delay. Wrong
nameservers in /etc/resolv.conf.
But it's always nice to have some tweak advice, so here are some info.

storeconfigs is off.
foreman uses mysql backend.
currently there are 25 nodes in foreman/puppet.

[root@deployment ~]# cat /etc/httpd/conf.d/passenger.conf
LoadModule passenger_module
/usr/lib/ruby/gems/1.8/gems/passenger-3.0.9/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9
PassengerRuby /usr/bin/ruby

PassengerHighPerformance On
PassengerMaxRequests 1000
PassengerMaxPoolSize 40
PassengerPoolIdleTime 240
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect On
PassengerUseGlobalQueue On

[root@deployment ~]# cat /etc/httpd/conf.d/foreman.conf
<VirtualHost *:80>
ServerName deployment
DocumentRoot /usr/share/foreman/public
PassengerAppRoot /usr/share/foreman

RailsAutoDetect On
AddDefaultCharset UTF-8

</VirtualHost>

<VirtualHost 84.17.203.72:443>
ServerName deployment

RailsAutoDetect On
DocumentRoot /usr/share/foreman/public
PassengerAppRoot /usr/share/foreman

SSLEngine On
— snip —
bunch of ssl cert options.
— snip —
</VirtualHost>

[root@deployment ~]# time curl
http://deployment/node/vguest03.unixwarez.org?format=yml

··· --- parameters: puppetmaster: deployment foreman_env: &id001 production owner_email: myemail@address.com owner_name: Admin User root_pw: xxxxxxxxxx environment: *id001 classes: [] real 0m0.031s user 0m0.001s sys 0m0.010s

What does it mean that the puppet master is drifting?
Also there are no swapping.

2012/5/11 Ohad Levy ohadlevy@gmail.com

ב-11 במאי 2012, בשעה 01:16, Elias Abacioglu elias.rabi@gmail.com כתב/ה:

Hi,

I’ve recently suffered from a really annoying problem.

“new nodes did not register properly with puppet” or at least that was
my description.
So old nodes worked good, new nodes had no catalog. After a lot of
digging I found out that the ENC was not working and problem was it got
timed out against the URL(maybe there should be a syslog error when that
happens?) and it started using the cached yaml files instead.
So this issue appeared after I did a yum upgrade on my CentOS 6. Somehow
the updates caused foreman to respond slower than before. So the timeout
that was set to 3 seconds always hit and I was force to set it to 10
seconds. After that it worked.
I’ve also buffed my virtual server from 1 vCPU to 2 vCPU and from 1GB
RAM to 4GB RAM. Still curl against foreman takes 5 seconds to get the yaml
output.

So after this horrific story… I actually have a question. I run puppet
master + apache + passenger + foreman + mysql on the same server. Does any
one have a performance tweak guide? Before I have only followed the
passenger tweaks that are on puppetlabs wiki…

It’s hard to say why (from the description) but 5 seconds is not
acceptable.
Can you describe your setup? Is your machine swapping a lot? If you could
go over the logs and see what takes the longest that might be a starting
point.

Ohad

Regards,
/Raboo

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to
foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/foreman-users?hl=en.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to
foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/foreman-users?hl=en.

5

> Hi,
>
> After more digging I was able to get to the root cause of the delay. Wrong nameservers in /etc/resolv.conf.
> But it's always nice to have some tweak advice, so here are some info.

Thats great. I imagine the response is pretty quick now.
>
> storeconfigs is off.
> foreman uses mysql backend.
> currently there are 25 nodes in foreman/puppet.

No need to separate your foreman server from the puppet master unless you plan on expanding to 99+ nodes.

>
>
> [root@deployment ~]# cat /etc/httpd/conf.d/passenger.conf
> LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9/ext/apache2/mod_passenger.so
> PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9
> PassengerRuby /usr/bin/ruby
>
> PassengerHighPerformance On
> PassengerMaxRequests 1000
> PassengerMaxPoolSize 40
> PassengerPoolIdleTime 240
> PassengerStatThrottleRate 120
> RackAutoDetect Off
> RailsAutoDetect On
> PassengerUseGlobalQueue On
>
>
> [root@deployment ~]# cat /etc/httpd/conf.d/foreman.conf
> <VirtualHost *:80>
> ServerName deployment
> DocumentRoot /usr/share/foreman/public
> PassengerAppRoot /usr/share/foreman
>
> RailsAutoDetect On
> AddDefaultCharset UTF-8
>
> </VirtualHost>
>
> <VirtualHost 84.17.203.72:443>
> ServerName deployment
>
> RailsAutoDetect On
> DocumentRoot /usr/share/foreman/public
> PassengerAppRoot /usr/share/foreman
>
> SSLEngine On
> — snip —
> bunch of ssl cert options.
> — snip —
> </VirtualHost>
>
> [root@deployment ~]# time curl http://deployment/node/vguest03.unixwarez.org?format=yml
> —
> parameters:
> puppetmaster: deployment
> foreman_env: &id001 production
> owner_email: myemail@address.com
> owner_name: Admin User
> root_pw: xxxxxxxxxx
> environment: *id001
> classes: []
> real 0m0.031s
> user 0m0.001s
> sys 0m0.010s
>
>
> What does it mean that the puppet master is drifting?

This has to do with the time on the puppet master. If the time is out of whack (drifted to far) puppet will start failing and the nodes will report that they couldn't authenticate to the puppet master. Of course you might see other things go wrong as well.
The Virtual machines hardware clock always seems to drift pretty far out when the VM host datastore has a lot of heavy IO from other VMs on the same datastore.
The time could also drift on the puppet client as well, so make sure they are always in sync with the vm host.

http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf (really good technical read)
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/chap-Virtualization-KVM_guest_timing_management.html

Corey

··· On May 11, 2012, at 1:58 AM, Elias Abacioglu wrote: > Also there are no swapping. > > 2012/5/11 Ohad Levy > > > ב-11 במאי 2012, בשעה 01:16, Elias Abacioglu כתב/ה: > > > Hi, > > > > I've recently suffered from a really annoying problem. > > > > "new nodes did not register properly with puppet" or at least that was my description. > > So old nodes worked good, new nodes had no catalog. After a lot of digging I found out that the ENC was not working and problem was it got timed out against the URL(maybe there should be a syslog error when that happens?) and it started using the cached yaml files instead. > > So this issue appeared after I did a yum upgrade on my CentOS 6. Somehow the updates caused foreman to respond slower than before. So the timeout that was set to 3 seconds always hit and I was force to set it to 10 seconds. After that it worked. > > I've also buffed my virtual server from 1 vCPU to 2 vCPU and from 1GB RAM to 4GB RAM. Still curl against foreman takes 5 seconds to get the yaml output. > > > > So after this horrific story.. I actually have a question. I run puppet master + apache + passenger + foreman + mysql on the same server. Does any one have a performance tweak guide? Before I have only followed the passenger tweaks that are on puppetlabs wiki.. > > > It's hard to say why (from the description) but 5 seconds is not acceptable. > Can you describe your setup? Is your machine swapping a lot? If you could go over the logs and see what takes the longest that might be a starting point. > > Ohad > > Regards, > > /Raboo > > > > -- > > You received this message because you are subscribed to the Google Groups "Foreman users" group. > > To post to this group, send email to foreman-users@googlegroups.com. > > To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com. > > For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To post to this group, send email to foreman-users@googlegroups.com. > To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To post to this group, send email to foreman-users@googlegroups.com. > To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.