Bug in stats and host status?

I think I have discovered a new bug where a host can have most recent
report that is in error state and active state, but not showing as error.

Any ideas why a host that has a report that has errors and changes, would
not show up in a search for all hosts with errors? Search: status.failed >
0 and status.enabled = true

(Yes the host in question is enabled.)

Initially I thought it was the following, and was going to write the
following, but digging further (using simplified searches) it appears to be
a new bug:

So it looks like the changes I committed to address
Bug #1617: Dashboard piechart stats don't match expectations. - Foreman have been completely refactored (yay!),
is at least in one case regressed (boo!)

The case where a host has an error and an active change in the same report
isn't making "error" the host's state. (It was agreed in an older mailing
list thread that error state should take higher precedent over active
state). The host state icon (red err), the dashboard stats/chart, etc,
should all reflect that the host is in error state. I know this worked as a
result of my that were committed, but at some point stopped working. (I am
running cc7c19279ad3393a5b5d60a8abe3e5abea494522 (1.1RC3) I don't know
when this stopped working as this isn't a very common edge case I run into.

This should be the commit where i fixed it the first time
https://github.com/theforeman/foreman/commit/0bb7ac8f3f071543bcb52c2c214708d8ebfb21a8

The code seems to have been refactored to the point, where I couldn't
quickly find where this change might have taken place, and I didn't really
have time to dig through commit logs/diffs. (Looks like the dashboard stuff
was largely moved into the model, but there are a bizillion commits on the
dashboard model.

Hoping this might ring some bells with folks, and if someone thinks this
was fixed since rc3 was cut, I will update and retest. If not, any other
thoughts on what might be going on? (I have the edge case in place, and can
hold off resolving it, if someone needs to see it in the wild.)

Thanks,
Brian

··· --

OK a little further digging. It appears that there is an error in the
reports that isn't getting flagged as an error. It is effecting many hosts.

This is the sanitized error:

Could not prefetch ssh_authorized_key provider 'parsed': Could not parse
line "ssh-dss KEYNAME@BLAH" at /home/username/.ssh/authorized_keys:3

I'm not sure what the next steps here would be to figure out why it's not
getting flagged as an error state, even though it shows up as an error in
the reports.

Thanks,
Brian

··· On Fri, Jan 4, 2013 at 6:20 PM, Brian Gupta wrote:

I think I have discovered a new bug where a host can have most recent
report that is in error state and active state, but not showing as error.

Any ideas why a host that has a report that has errors and changes, would
not show up in a search for all hosts with errors? Search: status.failed >
0 and status.enabled = true

(Yes the host in question is enabled.)

Initially I thought it was the following, and was going to write the
following, but digging further (using simplified searches) it appears to be
a new bug:

So it looks like the changes I committed to address
Bug #1617: Dashboard piechart stats don't match expectations. - Foreman have been completely refactored (yay!),
is at least in one case regressed (boo!)

The case where a host has an error and an active change in the same report
isn’t making “error” the host’s state. (It was agreed in an older mailing
list thread that error state should take higher precedent over active
state). The host state icon (red err), the dashboard stats/chart, etc,
should all reflect that the host is in error state. I know this worked as a
result of my that were committed, but at some point stopped working. (I am
running cc7c19279ad3393a5b5d60a8abe3e5abea494522 (1.1RC3) I don’t know
when this stopped working as this isn’t a very common edge case I run into.

This should be the commit where i fixed it the first time
https://github.com/theforeman/foreman/commit/0bb7ac8f3f071543bcb52c2c214708d8ebfb21a8

The code seems to have been refactored to the point, where I couldn’t
quickly find where this change might have taken place, and I didn’t really
have time to dig through commit logs/diffs. (Looks like the dashboard stuff
was largely moved into the model, but there are a bizillion commits on the
dashboard model.

Hoping this might ring some bells with folks, and if someone thinks this
was fixed since rc3 was cut, I will update and retest. If not, any other
thoughts on what might be going on? (I have the edge case in place, and can
hold off resolving it, if someone needs to see it in the wild.)

Thanks,
Brian


http://aws.amazon.com/solutions/solution-providers/brandorr/


http://aws.amazon.com/solutions/solution-providers/brandorr/

> OK a little further digging. It appears that there is an error in the
> reports that isn't getting flagged as an error. It is effecting many hosts.
>
> This is the sanitized error:
>
> Could not prefetch ssh_authorized_key provider 'parsed': Could not parse
> line "ssh-dss KEYNAME@BLAH" at /home/username/.ssh/authorized_keys:3
>
> I'm not sure what the next steps here would be to figure out why it's not
> getting flagged as an error state, even though it shows up as an error in
> the reports.
>
> which version of puppet is it? older than 2.6.5?

Ohad

··· On Sat, Jan 5, 2013 at 1:36 AM, Brian Gupta wrote:

Thanks,
Brian

On Fri, Jan 4, 2013 at 6:20 PM, Brian Gupta brian.gupta@brandorr.comwrote:

I think I have discovered a new bug where a host can have most recent
report that is in error state and active state, but not showing as error.

Any ideas why a host that has a report that has errors and changes, would
not show up in a search for all hosts with errors? Search: status.failed >
0 and status.enabled = true

(Yes the host in question is enabled.)

Initially I thought it was the following, and was going to write the
following, but digging further (using simplified searches) it appears to be
a new bug:

So it looks like the changes I committed to address
Bug #1617: Dashboard piechart stats don't match expectations. - Foreman have been completely refactored
(yay!), is at least in one case regressed (boo!)

The case where a host has an error and an active change in the same
report isn’t making “error” the host’s state. (It was agreed in an older
mailing list thread that error state should take higher precedent over
active state). The host state icon (red err), the dashboard stats/chart,
etc, should all reflect that the host is in error state. I know this worked
as a result of my that were committed, but at some point stopped working.
(I am running cc7c19279ad3393a5b5d60a8abe3e5abea494522 (1.1RC3) I don’t
know when this stopped working as this isn’t a very common edge case I run
into.

This should be the commit where i fixed it the first time
https://github.com/theforeman/foreman/commit/0bb7ac8f3f071543bcb52c2c214708d8ebfb21a8

The code seems to have been refactored to the point, where I couldn’t
quickly find where this change might have taken place, and I didn’t really
have time to dig through commit logs/diffs. (Looks like the dashboard stuff
was largely moved into the model, but there are a bizillion commits on the
dashboard model.

Hoping this might ring some bells with folks, and if someone thinks this
was fixed since rc3 was cut, I will update and retest. If not, any other
thoughts on what might be going on? (I have the edge case in place, and can
hold off resolving it, if someone needs to see it in the wild.)

Thanks,
Brian


http://aws.amazon.com/solutions/solution-providers/brandorr/


http://aws.amazon.com/solutions/solution-providers/brandorr/

>
>
>
>
>> OK a little further digging. It appears that there is an error in the
>> reports that isn't getting flagged as an error. It is effecting many hosts.
>>
>> This is the sanitized error:
>>
>> Could not prefetch ssh_authorized_key provider 'parsed': Could not parse
>> line "ssh-dss KEYNAME@BLAH" at /home/username/.ssh/authorized_keys:3
>>
>> I'm not sure what the next steps here would be to figure out why it's not
>> getting flagged as an error state, even though it shows up as an error in
>> the reports.
>>
>> which version of puppet is it? older than 2.6.5?
>

2.6.9.

··· On Sat, Jan 5, 2013 at 7:33 AM, Ohad Levy wrote: > On Sat, Jan 5, 2013 at 1:36 AM, Brian Gupta wrote:

Ohad

Thanks,
Brian

On Fri, Jan 4, 2013 at 6:20 PM, Brian Gupta brian.gupta@brandorr.comwrote:

I think I have discovered a new bug where a host can have most recent
report that is in error state and active state, but not showing as error.

Any ideas why a host that has a report that has errors and changes,
would not show up in a search for all hosts with errors? Search:
status.failed > 0 and status.enabled = true

(Yes the host in question is enabled.)

Initially I thought it was the following, and was going to write the
following, but digging further (using simplified searches) it appears to be
a new bug:

So it looks like the changes I committed to address
Bug #1617: Dashboard piechart stats don't match expectations. - Foreman have been completely refactored
(yay!), is at least in one case regressed (boo!)

The case where a host has an error and an active change in the same
report isn’t making “error” the host’s state. (It was agreed in an older
mailing list thread that error state should take higher precedent over
active state). The host state icon (red err), the dashboard stats/chart,
etc, should all reflect that the host is in error state. I know this worked
as a result of my that were committed, but at some point stopped working.
(I am running cc7c19279ad3393a5b5d60a8abe3e5abea494522 (1.1RC3) I don’t
know when this stopped working as this isn’t a very common edge case I run
into.

This should be the commit where i fixed it the first time
https://github.com/theforeman/foreman/commit/0bb7ac8f3f071543bcb52c2c214708d8ebfb21a8

The code seems to have been refactored to the point, where I couldn’t
quickly find where this change might have taken place, and I didn’t really
have time to dig through commit logs/diffs. (Looks like the dashboard stuff
was largely moved into the model, but there are a bizillion commits on the
dashboard model.

Hoping this might ring some bells with folks, and if someone thinks this
was fixed since rc3 was cut, I will update and retest. If not, any other
thoughts on what might be going on? (I have the edge case in place, and can
hold off resolving it, if someone needs to see it in the wild.)

Thanks,
Brian


http://aws.amazon.com/solutions/solution-providers/brandorr/


http://aws.amazon.com/solutions/solution-providers/brandorr/


http://aws.amazon.com/solutions/solution-providers/brandorr/

Wondering if this is related to the recent commit where we magically detect
the report version? Is there a way to test if Foreman is detecting
properly? Or any other next steps to figure out what is going on here?

Thanks,
Brian

··· On Tue, Jan 8, 2013 at 3:50 PM, Brian Gupta wrote:

On Sat, Jan 5, 2013 at 7:33 AM, Ohad Levy ohadlevy@gmail.com wrote:

On Sat, Jan 5, 2013 at 1:36 AM, Brian Gupta brian.gupta@brandorr.comwrote:

OK a little further digging. It appears that there is an error in the
reports that isn’t getting flagged as an error. It is effecting many hosts.

This is the sanitized error:

Could not prefetch ssh_authorized_key provider ‘parsed’: Could not parse
line “ssh-dss KEYNAME@BLAH” at /home/username/.ssh/authorized_keys:3

I’m not sure what the next steps here would be to figure out why it’s
not getting flagged as an error state, even though it shows up as an error
in the reports.

which version of puppet is it? older than 2.6.5?

2.6.9.