Hi,
Have attached all logs.
apache – ssl_accesslog
172.30.251.169 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.248.140?format=yml HTTP/1.1" 200 157 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.251.169 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.245.67?format=yml HTTP/1.1" 200 151 "-" "-"
172.30.251.169 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.251.133?format=yml HTTP/1.1" 200 143 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "POST /api/hosts/facts
HTTP/1.1" 201 899 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "POST /api/hosts/facts
HTTP/1.1" 201 902 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.253.227?format=yml HTTP/1.1" 200 157 "-" "-"
172.30.251.169 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.250.251?format=yml HTTP/1.1" 200 169 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "POST /api/hosts/facts
HTTP/1.1" 201 896 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.245.37?format=yml HTTP/1.1" 200 143 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "POST /api/hosts/facts
HTTP/1.1" 201 899 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.244.64?format=yml HTTP/1.1" 200 156 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 572 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "POST /api/hosts/facts
HTTP/1.1" 201 899 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.251.163?format=yml HTTP/1.1" 200 157 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "GET
/node/172.30.253.191?format=yml HTTP/1.1" 200 157 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.252.254 - - [14/Jul/2014:03:01:13 -0700] "POST /api/reports
HTTP/1.1" 201 573 "-" "-"
172.30.244.56 - - [14/Jul/2014:03:01:13 -0700] "POST /api/hosts/facts
HTTP/1.1" 201 902 "-" "-"
172.30.251.169 - - [14/Jul/2014:03:01:13 -0700] "POST /api/hosts/facts
HTTP/1.1" 201 899 "-" "-"
apache erro log
18824 stdout]
[ 2014-07-14 03:01:18.4136 29604/7f6c94794700 Pool2/SmartSpawner.h:301 ]:
Preloader for /usr/share/foreman started on PID 18824, listening on
unix:/tmp/passenger.1.0.29595/generation-0/backends/preloader.18837
foreman log:
Started GET "/node/172.30.251.147?format=yml" for 172.30.251.169 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.251.147"}
Rendered text template (0.0ms)
Completed 200 OK in 179ms (Views: 0.8ms | ActiveRecord: 2.6ms)
Rendered text template (0.0ms)
Completed 200 OK in 54ms (Views: 0.7ms | ActiveRecord: 2.5ms)
Started GET "/node/172.30.254.225?format=yml" for 172.30.244.56 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.254.225"}
Started GET "/node/172.30.251.181?format=yml" for 172.30.244.56 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.251.181"}
Rendered text template (0.0ms)
Completed 200 OK in 52ms (Views: 0.6ms | ActiveRecord: 2.6ms)
Started GET "/node/172.30.253.245?format=yml" for 172.30.244.56 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.253.245"}
Rendered text template (0.0ms)
Completed 200 OK in 70ms (Views: 0.9ms | ActiveRecord: 3.1ms)
Started GET "/node/172.30.251.148?format=yml" for 172.30.244.56 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.251.148"}
Rendered text template (0.0ms)
Completed 200 OK in 55ms (Views: 0.8ms | ActiveRecord: 3.0ms)
Started GET "/node/172.30.253.227?format=yml" for 172.30.244.56 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.253.227"}
Rendered text template (0.0ms)
Completed 200 OK in 70ms (Views: 1.0ms | ActiveRecord: 2.7ms)
Started GET "/node/172.30.252.180?format=yml" for 172.30.251.169 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.252.180"}
Rendered text template (0.0ms)
Completed 200 OK in 60ms (Views: 0.8ms | ActiveRecord: 3.0ms)
Started GET "/node/172.30.251.114?format=yml" for 172.30.251.169 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.251.114"}
Rendered text template (0.0ms)
Completed 200 OK in 61ms (Views: 0.7ms | ActiveRecord: 2.6ms)
Started GET "/node/172.30.254.156?format=yml" for 172.30.244.56 at
2014-07-14 03:01:03 -0700
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"172.30.254.156"}
Rendered text template (0.0ms)
Completed 200 OK in 185ms (Views: 0.7ms | ActiveRecord: 2.6ms)
Rendered text template (0.0ms)
Completed 200 OK in 58ms (Views: 0.9ms | ActiveRecord: 2.7ms)
Regards,
Suresh
···
On Monday, 14 July 2014 16:11:52 UTC+5:30, ohad wrote:
>
>
>
>
> On Mon, Jul 14, 2014 at 1:13 PM, Suresh P > wrote:
>
>> Hi,
>>
>> I'm running 3 puppet masters(16 core cpu, 32GB RAM) under lvs and using
>> foreman server(16 core cpu, 24GB RAM) for ENC and Reports. In my
>> infrastructure i have running puppet agent in totally 660 production nodes.
>>
>> Following are foreman and puppet versions.
>> foreman version 1.5.1
>> puppet server version 3.6.2
>>
>> My issue is:
>> 1. Most of the time around 35 puppet agent nodes going "out of sync", in
>> this node i'm getting following error message.
>>
>> Jul 14 03:01:16 172 puppet-agent[7652]: Could not retrieve catalog from
>>> remote server: Error 503 on SERVER: This website is under heavy
>>> load
We're sorry, too many people are accessing this website at the
>>> same time. We're working on this problem. Please try again later.
>>> Jul 14 03:01:16 172 puppet-agent[7652]: Using cached catalog
>>> Jul 14 03:01:17 172 puppet-agent[7652]: hello from puppet
>>> Jul 14 03:01:17 172 puppet-agent[7652]: (/Stage[main]/Main/Notify[hello
>>> from puppet]/message) defined 'message' as 'hello from puppet'
>>> Jul 14 03:01:17 172 puppet-agent[7652]: Finished catalog run in 0.04
>>> seconds
>>> Jul 14 03:01:17 172 puppet-agent[7652]: Could not send report: Error 503
>>> on SERVER: This website is under heavy load
We're sorry, too
>>> many people are accessing this website at the same time. We're working on
>>> this problem. Please try again later.
>>>
>>
>> 2. Around 20-25 puppet agent nodes is in "Host in error state", on that
>> nodes getting following error.
>>
>> Jul 14 03:01:12 172 puppet-agent[20034]: Could not retrieve catalog from
>>> remote server: Error 503 on SERVER: This website is under heavy
>>> load
We're sorry, too many people are accessing this website at the
>>> same time. We're working on this problem. Please try again later.
>>> Jul 14 03:01:13 172 puppet-agent[20034]: Using cached catalog
>>> Jul 14 03:01:13 172 puppet-agent[20034]: hello from puppet
>>> Jul 14 03:01:13 172 puppet-agent[20034]: (/Stage[main]/Main/Notify[hello
>>> from puppet]/message) defined 'message' as 'hello from puppet'
>>> Jul 14 03:01:13 172 puppet-agent[20034]: Finished catalog run in 0.35
>>> seconds
>>>
>>
>> Can you please provide me the solution to come out from this error.
>> Thanks in advance for your support.
>>
>
> Can you provide the apache logs or foreman logs for the same time. to me,
> it sounds like a passenger configuration that you can tweak.
>
> Ohad
>
>>
>> Regards,
>> Suresh
>>
>>
>>
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Foreman users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-user...@googlegroups.com .
>> To post to this group, send email to forema...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/foreman-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>