I'm having an issue with reports for certain hosts coming back
blank…puppet agent runs on the clients seem to be finishing just fine,
but in Foreman, many of the reports are blank and thus show the hosts in an
error state. For the majority of hosts, everything is working as expected.
On the master server, I'm getting the following log message relatively
frequently, which might be the cause?
Apr 2 11:53:08 puppet puppet-master[16141]: Report http failed:
Connection refused - connect(2)
It's not being blocked by iptables or selinux, and the failures seem
intermittent, but mostly limited to hosts in a particular data center (~25%
of my hosts have this issue pretty consistently). The master server is
running phusion passenger for both puppetmasterd and foreman:
[root@puppet ~]# passenger-status
----------- General information -----------
max = 30
count = 12
active = 1
inactive = 11
Waiting on global queue: 0
----------- Domains -----------
/usr/share/foreman:
PID: 13630 Sessions: 0 Processed: 32 Uptime: 8m 45s
PID: 13230 Sessions: 0 Processed: 66 Uptime: 24m 20s
/usr/share/puppet/rack/puppetmasterd:
PID: 15938 Sessions: 0 Processed: 203 Uptime: 7m 9s
PID: 16141 Sessions: 0 Processed: 758 Uptime: 7m 2s
PID: 2718 Sessions: 0 Processed: 1240 Uptime: 13m 42s
PID: 2692 Sessions: 0 Processed: 566 Uptime: 13m 44s
PID: 15954 Sessions: 0 Processed: 271 Uptime: 7m 7s
PID: 15839 Sessions: 0 Processed: 105 Uptime: 7m 16s
PID: 16066 Sessions: 0 Processed: 2 Uptime: 7m 5s
PID: 15889 Sessions: 0 Processed: 2 Uptime: 7m 12s
PID: 15870 Sessions: 0 Processed: 2 Uptime: 7m 14s
PID: 16212 Sessions: 1 Processed: 1186 Uptime: 7m 0s
For whatever reason, I never see more than two processes for the Foreman
app when checking passenger-status. Beyond that, I'm not finding any reason
that the reports would be failing, and nothing too enlightening in the
logs. As another twist, when I log on to the hosts with blank reports and
run puppet agent --test, the reports usually come through just fine. Is
there something obvious I may be missing here?
I've attached two pictures, one of a host displaying a blank report, and
then the report history for the same host, where you can see one successful
report, and then failures (which were all blank)…nothing was done on the
host during that time.
Its hard to tell from the info you provided…
http error seems to be non related to foreman, do you have a http
report enabled in your master puppet.conf?
also, can you see the actual puppet output on "failed" runs? (e.g. on
the host syslog).
and last, maybe you should also consider upgrading, as the version you
are running is about 1 year old.
Ohad
···
On Mon, Apr 2, 2012 at 8:24 PM, Aaron Bull Schaefer wrote:
> I'm having an issue with reports for certain hosts coming back
> blank...puppet agent runs on the clients seem to be finishing just fine, but
> in Foreman, many of the reports are blank and thus show the hosts in an
> error state. For the majority of hosts, everything is working as expected.
>
> On the master server, I'm getting the following log message relatively
> frequently, which might be the cause?
>
> Apr 2 11:53:08 puppet puppet-master[16141]: Report http failed:
> Connection refused - connect(2)
>
> It's not being blocked by iptables or selinux, and the failures seem
> intermittent, but mostly limited to hosts in a particular data center (~25%
> of my hosts have this issue pretty consistently). The master server is
> running phusion passenger for both puppetmasterd and foreman:
>
> [root@puppet ~]# passenger-status
> ----------- General information -----------
> max = 30
> count = 12
> active = 1
> inactive = 11
> Waiting on global queue: 0
>
> ----------- Domains -----------
> /usr/share/foreman:
> PID: 13630 Sessions: 0 Processed: 32 Uptime: 8m 45s
> PID: 13230 Sessions: 0 Processed: 66 Uptime: 24m 20s
>
> /usr/share/puppet/rack/puppetmasterd:
> PID: 15938 Sessions: 0 Processed: 203 Uptime: 7m 9s
> PID: 16141 Sessions: 0 Processed: 758 Uptime: 7m 2s
> PID: 2718 Sessions: 0 Processed: 1240 Uptime: 13m 42s
> PID: 2692 Sessions: 0 Processed: 566 Uptime: 13m 44s
> PID: 15954 Sessions: 0 Processed: 271 Uptime: 7m 7s
> PID: 15839 Sessions: 0 Processed: 105 Uptime: 7m 16s
> PID: 16066 Sessions: 0 Processed: 2 Uptime: 7m 5s
> PID: 15889 Sessions: 0 Processed: 2 Uptime: 7m 12s
> PID: 15870 Sessions: 0 Processed: 2 Uptime: 7m 14s
> PID: 16212 Sessions: 1 Processed: 1186 Uptime: 7m 0s
>
> For whatever reason, I never see more than two processes for the Foreman app
> when checking passenger-status. Beyond that, I'm not finding any reason that
> the reports would be failing, and nothing too enlightening in the logs. As
> another twist, when I log on to the hosts with blank reports and run `puppet
> agent --test`, the reports usually come through just fine. Is there
> something obvious I may be missing here?
>
> I've attached two pictures, one of a host displaying a blank report, and
> then the report history for the same host, where you can see one successful
> report, and then failures (which were all blank)...nothing was done on the
> host during that time.
You were right, the http error was unrelated…a co-worker had temporarily
put http reports into place, and I didn't notice. So looking at the syslog
for the failed runs, it actually does look like there's a problem. On the
handful I checked, I see:
Could not run Puppet configuration client: Could not retrieve local
facts: No route to host - connect(2)
Manual runs indicated that it was a problem with the ec2.rb facter lib,
which led me to this bug report: http://projects.puppetlabs.com/issues/4526
Some of my machines were running a pre-1.5.8 version of Facter, so I've
updated those and also noticed that we were pushing out an old version of
that specific lib, so hopefully fixing that will clear things up. I'll post
again with more details if I'm still getting blank reports after sorting
all of this out. Thanks for the help!
···
On Monday, April 2, 2012 12:56:00 PM UTC-7, ohad wrote:
>
> Its hard to tell from the info you provided...
> http error seems to be non related to foreman, do you have a http
> report enabled in your master puppet.conf?
>
> also, can you see the actual puppet output on "failed" runs? (e.g. on
> the host syslog).
>
> and last, maybe you should also consider upgrading, as the version you
> are running is about 1 year old.
>
