Notifications / monitoring for infra

Lukas asked me to raise this after Koji ran out of space, so ... :)

Currently there are two forms of notification from our infra. Firstly there is the private infra Google Group, but no-one seems to know who maintains is (Dominic?). Secondly, there is a list of email addresses written by Puppet to /etc/aliases (current it's me, Ohad and Michael).

While notifications from Koji would have at least reached more people if it was in Puppet (which we discussed briefly, but no one had time for), I think it makes sense to think about what we want to do more generally about this stuff.

My *long term* preference would be proper monitoring - something like Icinga or Zabbix, properly deployed with the appropriate checks, and the ability to alert the right people if a check fails. That's the right thing to do, and is flexibile in who gets a particular alert.

In the short term, though, we could use either *another* mailing list (since the existing Google Group seems lost) or a private infra group on Discourse (I tested it briefly, you can email in to a group and keep the messages private to the group members). Either of these would work in the short term. We could also use an alias on the root DNS
(infra@theforeman.org or something) but I don't think Ohad can set up multiple forwards for that very easily... and it's harder to manage when the list of forwards changes (only Ohad can do it).

Thoughts? I lean to a temporary solution on Discourse (but not strongly, a mailing list is fine here), and proper monitoring to be added to Eric's infra roadmap for later on.

Greg
I don't really care as long as you give us an email that we can configure both our puppet and non-puppet managed servers to send e-mails to and you add me and Eric into a private list/alias/RSS.

LZ


··· On Wed, Dec 6, 2017 at 12:40 AM, Greg Sutcliffe <greg@emeraldreverie.org> wrote:
Lukas asked me to raise this after Koji ran out of space, so ... :)

Currently there are two forms of notification from our infra. Firstly
there is the private infra Google Group, but no-one seems to know who
maintains is (Dominic?). Secondly, there is a list of email addresses
written by Puppet to /etc/aliases (current it's me, Ohad and Michael).

While notifications from Koji would have at least reached more people if
it was in Puppet (which we discussed briefly, but no one had time for),
I think it makes sense to think about what we want to do more generally
about this stuff.

My *long term* preference would be proper monitoring - something like
Icinga or Zabbix, properly deployed with the appropriate checks, and the
ability to alert the right people if a check fails. That's the right
thing to do, and is flexibile in who gets a particular alert.

In the short term, though, we could use either *another* mailing list
(since the existing Google Group seems lost) or a private infra group on
Discourse (I tested it briefly, you can email in to a group and keep the
messages private to the group members). Either of these would work in
the short term. We could also use an alias on the root DNS
(infra@theforeman.org or something) but I don't think Ohad can set up
multiple forwards for that very easily... and it's harder to manage when
the list of forwards changes (only Ohad can do it).

Thoughts? I lean to a temporary solution on Discourse (but not strongly,
a mailing list is fine here), and proper monitoring to be added to
Eric's infra roadmap for later on.

Greg

--
You received this message because you are subscribed to the Google Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal