ABRT plugin for Foreman

Hi!

I'm from the team that develops ABRT [1], the Automatic Bug Reporting Tool
that is currently used in Fedora and RH Enterprise Linux. ABRT can
detect several types of application problems (e.g. binary segfaults,
uncaught python exceptions, …), collect related data, and more or less
automatically report them somewhere.

For the detected crashes, ABRT can send something called uReport [2], a
small, machine-friendly report that doesn't necessarily contain enough
information to fix the underlying problem but can be useful to determine
whether similar problem happened before and to maintain crash
statistics. These reports can be collected by ABRT server (called "FAF"
in some places) - Fedora users send them to the instance located here
[3].

What we would like to do is to provide a Foreman plugin that would
associate to every Foreman-managed machine the uReports the machine
sent. The reports would then be displayed in the web interface so that
the administrator can see if something bad is going on on their machine.

We're not familiar with Foreman and still trying to figure out how would
this integration work. After a quick discussion with Foreman guys in Red
Hat Brno office, we've identified two design problems:

  1. How should Foreman and ABRT server interact
··· ----------------------------------------------

We were able to come up with two scenarios:

A) uReports are collected by ABRT server deployed by the administrator.
Upon receiving a report, the ABRT server notifies Foreman (or Foreman
can periodically ask the ABRT server for new reports). Foreman
communicates with ABRT server using some kind of REST API.

  • machine authentication has to be handled by ABRT server
  • ABRT server has to provide suitable API
  • not necessary to duplicate ABRT server code in Foreman

B) uReports are collected by Foreman, or some kind of proxy written for
this purpose. The reports can be browsed in Foreman and can be
forwarded to ABRT server instance, either automatically or after the
administrator manually accepts the report. The report can be
forwarded to ABRT server run by the administrator, or it can be
forwarded to the instance managed by Fedora/Red Hat/etc.

  • Foreman handles machine authentication
  • administrator can benefit from this without deploying their own ABRT
    server
  • subset of the ABRT server functionality would have to be implemented
    by the Foreman plugin/proxy
  1. Machine authentication

uReports were originally designed to allow anonymous reporting, mainly
for Fedora users. They only contain data that are not considered
sensitive and we currently have no way to find out where did an uReport
come from.

While we could just add an item containing e.g. FQDN to the uReport,
such information can be easily spoofed. Can we take advantage of the
fact that there already exists authentication between the managed
machines and Foreman (or Puppet?)?

I’ll be grateful for your thoughts on this subject.
Cheers,
Martin

[1] https://github.com/abrt
[2] https://github.com/abrt/faf/wiki/uReport#ureport2
[3] https://retrace.fedoraproject.org/faf/summary/

> Hi!
Hey,

> A) uReports are collected by ABRT server deployed by the administrator.
> Upon receiving a report, the ABRT server notifies Foreman (or Foreman
> can periodically ask the ABRT server for new reports). Foreman
> communicates with ABRT server using some kind of REST API.

We have nice Puppet based installer, so this looks like viable option to
me as we can setup another component.

> B) uReports are collected by Foreman, or some kind of proxy written for

I prefer A, this is solely my opinion.

> While we could just add an item containing e.g. FQDN to the uReport,
> such information can be easily spoofed. Can we take advantage of the
> fact that there already exists authentication between the managed
> machines and Foreman (or Puppet?)?

Someone correct me if I am wrong, but we are deploying Puppet client
certificate during provisioning phase which is being signed by Puppet CA
authority. That means each Foreman-managed machine has a client
certificate which could be re-used for other things. It should not be
a problem to use Puppet CA to validate client certificates during ABRT
upload.

The key is to make sure ABRT server has access to the Puppet CA
certificate (and key).

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

> Hi!
>
> I'm from the team that develops ABRT [1], the Automatic Bug Reporting Tool
> that is currently used in Fedora and RH Enterprise Linux. ABRT can
> detect several types of application problems (e.g. binary segfaults,
> uncaught python exceptions, …), collect related data, and more or less
> automatically report them somewhere.
>
> For the detected crashes, ABRT can send something called uReport [2], a
> small, machine-friendly report that doesn't necessarily contain enough
> information to fix the underlying problem but can be useful to determine
> whether similar problem happened before and to maintain crash
> statistics. These reports can be collected by ABRT server (called "FAF"
> in some places) - Fedora users send them to the instance located here
> [3].
>
> What we would like to do is to provide a Foreman plugin that would
> associate to every Foreman-managed machine the uReports the machine
> sent. The reports would then be displayed in the web interface so that
> the administrator can see if something bad is going on on their machine.
>
Great :slight_smile:

>
> We're not familiar with Foreman and still trying to figure out how would
> this integration work. After a quick discussion with Foreman guys in Red
> Hat Brno office, we've identified two design problems:
>
>
> 1. How should Foreman and ABRT server interact
> ----------------------------------------------
>
> We were able to come up with two scenarios:
>
> A) uReports are collected by ABRT server deployed by the administrator.
> Upon receiving a report, the ABRT server notifies Foreman (or Foreman
> can periodically ask the ABRT server for new reports). Foreman
> communicates with ABRT server using some kind of REST API.
>
> * machine authentication has to be handled by ABRT server
> * ABRT server has to provide suitable API
> * not necessary to duplicate ABRT server code in Foreman
>
Does it make sense to have multiple abrt servers? does it make sense to tie
these into foreman proxies (e.g. when having disconnected networks?)

>
> B) uReports are collected by Foreman, or some kind of proxy written for
> this purpose. The reports can be browsed in Foreman and can be
> forwarded to ABRT server instance, either automatically or after the
> administrator manually accepts the report. The report can be
> forwarded to ABRT server run by the administrator, or it can be
> forwarded to the instance managed by Fedora/Red Hat/etc.
>
> * Foreman handles machine authentication
> * administrator can benefit from this without deploying their own ABRT
> server
> * subset of the ABRT server functionality would have to be implemented
> by the Foreman plugin/proxy

Which level of complexity are we talking about here? does it make sense to
rewrite the abrt server? (or is it only a subset of the functionality?)

>

>
> 2. Machine authentication
> -------------------------
>
> uReports were originally designed to allow anonymous reporting, mainly
> for Fedora users. They only contain data that are not considered
> sensitive and we currently have no way to find out where did an uReport
> come from.
>
> While we could just add an item containing e.g. FQDN to the uReport,
> such information can be easily spoofed. Can we take advantage of the
> fact that there already exists authentication between the managed
> machines and Foreman (or Puppet?)?
>

I would assume in this case, you would want your systems to be known, we
could utilize the system facts, or uuid to identify it.

Thanks for looking into this,
Ohad

··· On Wed, Dec 4, 2013 at 6:56 PM, Martin Milata wrote:

I’ll be grateful for your thoughts on this subject.
Cheers,
Martin

[1] https://github.com/abrt
[2] https://github.com/abrt/faf/wiki/uReport#ureport2
[3] https://retrace.fedoraproject.org/faf/summary/


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

>
>
>
>
> Hi!
>
> I'm from the team that develops ABRT [1], the Automatic Bug
> Reporting Tool
> that is currently used in Fedora and RH Enterprise Linux. ABRT can
> detect several types of application problems (e.g. binary segfaults,
> uncaught python exceptions, …), collect related data, and more or less
> automatically report them somewhere.
>
> For the detected crashes, ABRT can send something called uReport [2], a
> small, machine-friendly report that doesn't necessarily contain enough
> information to fix the underlying problem but can be useful to determine
> whether similar problem happened before and to maintain crash
> statistics. These reports can be collected by ABRT server (called "FAF"
> in some places) - Fedora users send them to the instance located here
> [3].
>
> What we would like to do is to provide a Foreman plugin that would
> associate to every Foreman-managed machine the uReports the machine
> sent. The reports would then be displayed in the web interface so that
> the administrator can see if something bad is going on on their machine.
>
> Great :slight_smile:
>
>
> We're not familiar with Foreman and still trying to figure out how would
> this integration work. After a quick discussion with Foreman guys in Red
> Hat Brno office, we've identified two design problems:
>
>
> 1. How should Foreman and ABRT server interact
> ----------------------------------------------
>
> We were able to come up with two scenarios:
>
> A) uReports are collected by ABRT server deployed by the administrator.
> Upon receiving a report, the ABRT server notifies Foreman (or
> Foreman
> can periodically ask the ABRT server for new reports). Foreman
> communicates with ABRT server using some kind of REST API.
>
> * machine authentication has to be handled by ABRT server
> * ABRT server has to provide suitable API
> * not necessary to duplicate ABRT server code in Foreman
>
> Does it make sense to have multiple abrt servers? does it make sense to
> tie these into foreman proxies (e.g. when having disconnected networks?)
>
>
> B) uReports are collected by Foreman, or some kind of proxy written for
> this purpose. The reports can be browsed in Foreman and can be
> forwarded to ABRT server instance, either automatically or after the
> administrator manually accepts the report. The report can be
> forwarded to ABRT server run by the administrator, or it can be
> forwarded to the instance managed by Fedora/Red Hat/etc.
>
> * Foreman handles machine authentication
> * administrator can benefit from this without deploying their own ABRT
> server
> * subset of the ABRT server functionality would have to be implemented
> by the Foreman plugin/proxy
>
>
> Which level of complexity are we talking about here? does it make sense
> to rewrite the abrt server? (or is it only a subset of the functionality?)

Also… how did spacewalk handle this? I assume we would like to see a
similar model. When I first heard of this, I was assuming that foreman
contained the uReports.

>
>
>
> 2. Machine authentication
> -------------------------
>
> uReports were originally designed to allow anonymous reporting, mainly
> for Fedora users. They only contain data that are not considered
> sensitive and we currently have no way to find out where did an uReport
> come from.
>
> While we could just add an item containing e.g. FQDN to the uReport,
> such information can be easily spoofed. Can we take advantage of the
> fact that there already exists authentication between the managed
> machines and Foreman (or Puppet?)?
>
>
> I would assume in this case, you would want your systems to be known, we
> could utilize the system facts, or uuid to identify it.
>
> Thanks for looking into this,
> Ohad
>
– bk

··· On 12/05/2013 08:37 AM, Ohad Levy wrote: > On Wed, Dec 4, 2013 at 6:56 PM, Martin Milata > wrote:

Also bear in mind that Foreman's report format is now pure JSON, so if you
can format the uReport in a way which Foreman will understand, it's a
single api call to add a report to a Host in Foreman.

There's also moves to subclass the report format (for Chef) so then a
simple plugin could define it's own report format and expose an API route
accepting reports.

Greg

··· On 5 December 2013 15:01, Bryan Kearney wrote:

Also… how did spacewalk handle this? I assume we would like to see a
similar model. When I first heard of this, I was assuming that foreman
contained the uReports.

On the other hand, we do not want to add and maintain Foreman format in
ABRT clients, no matter how easy this is, upgrading clients is more
difficult.

··· On Thu, Dec 05, 2013 at 03:10:32PM +0000, Greg Sutcliffe wrote: > Also bear in mind that Foreman's report format is now pure JSON, so if you > can format the uReport in a way which Foreman will understand, it's a > single api call to add a report to a Host in Foreman.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

[…]
> > A) uReports are collected by ABRT server deployed by the administrator.
> > Upon receiving a report, the ABRT server notifies Foreman (or Foreman
> > can periodically ask the ABRT server for new reports). Foreman
> > communicates with ABRT server using some kind of REST API.
> >
> > * machine authentication has to be handled by ABRT server
> > * ABRT server has to provide suitable API
> > * not necessary to duplicate ABRT server code in Foreman
> >
>
> Does it make sense to have multiple abrt servers? does it make sense to tie
> these into foreman proxies (e.g. when having disconnected networks?)

I don't think it does. One of the main features of the server is
grouping the reports that were probably caused by the same bug. I'd say
it's best to have just one server.

> > B) uReports are collected by Foreman, or some kind of proxy written for
> > this purpose. The reports can be browsed in Foreman and can be
> > forwarded to ABRT server instance, either automatically or after the
> > administrator manually accepts the report. The report can be
> > forwarded to ABRT server run by the administrator, or it can be
> > forwarded to the instance managed by Fedora/Red Hat/etc.
> >
> > * Foreman handles machine authentication
> > * administrator can benefit from this without deploying their own ABRT
> > server
> > * subset of the ABRT server functionality would have to be implemented
> > by the Foreman plugin/proxy
>
>
> Which level of complexity are we talking about here? does it make sense to
> rewrite the abrt server? (or is it only a subset of the functionality?)

The server is quite complex as it does e.g. the report clustering or
resolving function names in stacktraces. I don't think rewriting it
entirely is reasonable. However, I can imagine that implementing a tiny
subset that would allow the administrator to see that a crash happened
in program X (from package Y) Z times in some timeframe would still be
useful.

> > 2. Machine authentication
> > -------------------------
> >
> > uReports were originally designed to allow anonymous reporting, mainly
> > for Fedora users. They only contain data that are not considered
> > sensitive and we currently have no way to find out where did an uReport
> > come from.
> >
> > While we could just add an item containing e.g. FQDN to the uReport,
> > such information can be easily spoofed. Can we take advantage of the
> > fact that there already exists authentication between the managed
> > machines and Foreman (or Puppet?)?
> >
>
> I would assume in this case, you would want your systems to be known, we
> could utilize the system facts, or uuid to identify it.

I think that without this assumption there wouldn't be many reasons to
integrate abrt server with foreman, as opposed to just running the
standalone abrt server somewhere on the network.

Thanks for the response,
Martin

··· On Thu, Dec 05, 2013 at 15:37:42 +0200, Ohad Levy wrote:

I wonder if it would be possible to use these certificates without major
changes to Puppet. Or, whether the benefits of having authenticated
problem reports outweigh the risks of sharing the puppet certificates
with another component.

Thanks for the reply,
Martin

··· On Thu, Dec 05, 2013 at 12:24:17 +0100, Lukas Zapletal wrote: > > Hi! > Hey, > > > A) uReports are collected by ABRT server deployed by the administrator. > > Upon receiving a report, the ABRT server notifies Foreman (or Foreman > > can periodically ask the ABRT server for new reports). Foreman > > communicates with ABRT server using some kind of REST API. > > We have nice Puppet based installer, so this looks like viable option to > me as we can setup another component. > > > B) uReports are collected by Foreman, or some kind of proxy written for > > I prefer A, this is solely my opinion. > > > While we could just add an item containing e.g. FQDN to the uReport, > > such information can be easily spoofed. Can we take advantage of the > > fact that there already exists authentication between the managed > > machines and Foreman (or Puppet?)? > > Someone correct me if I am wrong, but we are deploying Puppet client > certificate during provisioning phase which is being signed by Puppet CA > authority. That means each Foreman-managed machine has a client > certificate which could be re-used for other things. It should not be > a problem to use Puppet CA to validate client certificates during ABRT > upload. > > The key is to make sure ABRT server has access to the Puppet CA > certificate (and key).

Agree… I had assumed a plugin would extend Foreman to accept abrt
uReport formats and store them.

The same plugin pattern could be used for SCAP reports as well.

– bk

··· On 12/06/2013 04:00 AM, Lukas Zapletal wrote: > On Thu, Dec 05, 2013 at 03:10:32PM +0000, Greg Sutcliffe wrote: >> Also bear in mind that Foreman's report format is now pure JSON, so if you >> can format the uReport in a way which Foreman will understand, it's a >> single api call to add a report to a Host in Foreman. > > On the other hand, we do not want to add and maintain Foreman format in > ABRT clients, no matter how easy this is, upgrading clients is more > difficult. >

Well the same way we deploy client certificate, we can also deploy
another ABRT certificate.

··· On Fri, Dec 06, 2013 at 05:20:40PM +0100, Martin Milata wrote: > I wonder if it would be possible to use these certificates without major > changes to Puppet. Or, whether the benefits of having authenticated > problem reports outweigh the risks of sharing the puppet certificates > with another component.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

We just implemented the ability to use client-side SSL/TLS
authentication when sending uReports. This makes using Puppet
certificates easy as everything is handled by libcurl and it appears
that SELinux already allows us to access them.

If I understand correctly the plugin has to provide a smart proxy to
accept the uReports since the managed hosts are not required to have
connectivity to the main Foreman server. The idea is that the proxy
receives the uReports via client-authenticated https where the Puppet CA
is used to validate the client's certificate. Do you see any problem
with this?

We'd like to make the ureport client configuration easy so that the path
to the Puppet certificate doesn't have to be specified explicitly. I'm
not familiar with Puppet so I'd like to ask if we can expect the client
certificate to be in /var/lib/puppet/ssl/certs/facter fqdn.pem
and the key in /var/lib/puppet/ssl/private_keys/facter fqdn.pem ?

Thank you,
Martin Milata

··· On Sat, Dec 07, 2013 at 11:39:31 +0100, Lukas Zapletal wrote: > On Fri, Dec 06, 2013 at 05:20:40PM +0100, Martin Milata wrote: > > I wonder if it would be possible to use these certificates without major > > changes to Puppet. Or, whether the benefits of having authenticated > > problem reports outweigh the risks of sharing the puppet certificates > > with another component. > > Well the same way we deploy client certificate, we can also deploy > another ABRT certificate.

hello,

You may be interested in puppet config print | grep …

Claer

··· On Mon, Jan 20 2014 at 24:16, Martin Milata wrote: > On Sat, Dec 07, 2013 at 11:39:31 +0100, Lukas Zapletal wrote: > > On Fri, Dec 06, 2013 at 05:20:40PM +0100, Martin Milata wrote: > > > I wonder if it would be possible to use these certificates without major > > > changes to Puppet. Or, whether the benefits of having authenticated > > > problem reports outweigh the risks of sharing the puppet certificates > > > with another component. > > > > Well the same way we deploy client certificate, we can also deploy > > another ABRT certificate. > > We just implemented the ability to use client-side SSL/TLS > authentication when sending uReports. This makes using Puppet > certificates easy as everything is handled by libcurl and it appears > that SELinux already allows us to access them. > > If I understand correctly the plugin has to provide a smart proxy to > accept the uReports since the managed hosts are not required to have > connectivity to the main Foreman server. The idea is that the proxy > receives the uReports via client-authenticated https where the Puppet CA > is used to validate the client's certificate. Do you see any problem > with this? > > We'd like to make the ureport client configuration easy so that the path > to the Puppet certificate doesn't have to be specified explicitly. I'm > not familiar with Puppet so I'd like to ask if we can expect the client > certificate to be in /var/lib/puppet/ssl/certs/`facter fqdn`.pem > and the key in /var/lib/puppet/ssl/private_keys/`facter fqdn`.pem ?

> > We'd like to make the ureport client configuration easy so that the path
> > to the Puppet certificate doesn't have to be specified explicitly. I'm
> > not familiar with Puppet so I'd like to ask if we can expect the client
> > certificate to be in /var/lib/puppet/ssl/certs/facter fqdn.pem
> > and the key in /var/lib/puppet/ssl/private_keys/facter fqdn.pem ?
>
> You may be interested in puppet config print | grep …

Yeah, we cannot be sure how users will be configuring their puppet
clients. The default configuration is really as follows:

[root@hp-dl380pgen8-02-vm-4 ~]# puppet config print | grep pem
hostcert = /var/lib/puppet/ssl/certs/hp-dl380pgen8-02-vm-4.lab.bos.redhat.com.pem
hostprivkey = /var/lib/puppet/ssl/private_keys/hp-dl380pgen8-02-vm-4.lab.bos.redhat.com.pem
hostpubkey = /var/lib/puppet/ssl/public_keys/hp-dl380pgen8-02-vm-4.lab.bos.redhat.com.pem

But users might re-configure their puppet at will.

I think we can add ABRT configuration support as a snippet into our
community templates, so we make sure it is configured correctly for Red
Hat compatible systems. Something like:

HOSTCERT=$(puppet config print hostcert)
cat >/etc/abrt.conf <<EOABRT
[configuration]
blah=blah
cert=$HOSTCERT
EOABRT

Does this work for you?

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Ah, thanks for the suggestion, I changed the code to use "puppet config"
to find the certificate and key.

The template will still probably be needed to configure the uReport
server url anyway, though.

Thank you,
Martin

··· On Tue, Jan 21, 2014 at 11:20:48 +0100, Lukas Zapletal wrote: > > > We'd like to make the ureport client configuration easy so that the path > > > to the Puppet certificate doesn't have to be specified explicitly. I'm > > > not familiar with Puppet so I'd like to ask if we can expect the client > > > certificate to be in /var/lib/puppet/ssl/certs/`facter fqdn`.pem > > > and the key in /var/lib/puppet/ssl/private_keys/`facter fqdn`.pem ? > > > > You may be interested in puppet config print | grep ... > > Yeah, we cannot be sure how users will be configuring their puppet > clients. The default configuration is really as follows: > > [root@hp-dl380pgen8-02-vm-4 ~]# puppet config print | grep pem > hostcert = /var/lib/puppet/ssl/certs/hp-dl380pgen8-02-vm-4.lab.bos.redhat.com.pem > hostprivkey = /var/lib/puppet/ssl/private_keys/hp-dl380pgen8-02-vm-4.lab.bos.redhat.com.pem > hostpubkey = /var/lib/puppet/ssl/public_keys/hp-dl380pgen8-02-vm-4.lab.bos.redhat.com.pem > > But users might re-configure their puppet at will. > > I think we can add ABRT configuration support as a snippet into our > community templates, so we make sure it is configured correctly for Red > Hat compatible systems. Something like: > > HOSTCERT=$(puppet config print hostcert) > cat >/etc/abrt.conf < [configuration] > blah=blah > cert=$HOSTCERT > EOABRT > > Does this work for you?

> Ah, thanks for the suggestion, I changed the code to use "puppet config"
> to find the certificate and key.

I think this is an overkill:

https://github.com/abrt/libreport/blob/master/src/plugins/ureport.c#L38

Foreman or Puppet can configure ABRT and uReport plugin easily. This
parsing was not necessary. I apologize to confuse you in the first
place.

Foreman folks - do we have a tasks for ABRT/uReport integration? So far,
I have been able to find only:

http://projects.theforeman.org/issues/4151

LZ

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

> I think this is an overkill:
>
> https://github.com/abrt/libreport/blob/master/src/plugins/ureport.c#L38
>
> Foreman or Puppet can configure ABRT and uReport plugin easily. This
> parsing was not necessary. I apologize to confuse you in the first
> place.

This is still valid point ^^^^

But ignore everything bellow ----

> Foreman folks - do we have a tasks for ABRT/uReport integration? So far,
> I have been able to find only:
>
> Feature #4151: Add STI to reports and allow for registering new report types - Foreman

I was late on e-mails. I see there is ongoing discussion now.

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman