Foreman-debug script

Development
1

Hello,

I am starting to work on a simple katello-foreman script. It will likely
be very similar to this one:

https://github.com/Katello/katello/blob/master/src/script/katello-debug

Main purpose: collect all relevant log files, remove all possible
passwords or secret keys, create a tarball and tell the user where to
send/upload. The same for relevant config files.

In the future, this can be easily integrated with ABRT, but the first
cut is (I think) no ABRT integration. I believe the integration
interface is ABRT Custom Hooks which is very straighforward to integrate

  • the script just need to have an option to create all reports/files in
    the current directory rather than a temp dir:

https://fedorahosted.org/abrt/wiki/AbrtConfiguration

User can initiate the report using the ABRT command line utility. Later
on, we can integrate Foreman to trigger ABRT report automatically on
exception or something like that - this is I guess out of scope of this
effort.

Now, I am going to collect these files (skipped is not found):

  • some typical information (cpu, memory, etc)

  • ip addresses, hostname

  • iptables output

  • DNS resolve: localhost, hostname

  • content of /etc/hosts

  • foreman rails log

  • proxy main log

  • dhcp, dns, tftp log

  • libvirt log

  • virsh list output (?)

  • foreman settings.yml and database.yml

  • postgresql configuration

  • proxy setting.yml

  • foreman database table "settings" (?)

  • puppet main configuration

  • puppetmasterd log

  • puppet logs (?)

  • puppet certificates (serial ids)

(The first part can be optionally disabled - this is for the ABRT mode
because ABRT already collects this.)

The idea is you can run this on any server (e.g. DHCP server, DNS
server) and logs are collected if the relevant files are found.

We have been pretty successful with katello-debug-certificates which
prints out serial ids of all certificates and their CAs. I will
incorporate the same behavior into foreman-debug directly.

Opinions? Additional items or ideas?

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

2

Thanks Lukas!

>
>> Hello,
>>
>> I am starting to work on a simple katello-foreman script. It will likely
>> be very similar to this one:
>>
>> https://github.com/Katello/katello/blob/master/src/script/katello-debug
>
>
> Awesome :slight_smile:
>
> Main purpose: collect all relevant log files, remove all possible
>> passwords or secret keys, create a tarball and tell the user where to
>> send/upload. The same for relevant config files.
>>
>
> Be sure to scrub sensitive data, or our users might be irritated :slight_smile:
>
>
>> In the future, this can be easily integrated with ABRT, but the first
>> cut is (I think) no ABRT integration. I believe the integration
>> interface is ABRT Custom Hooks which is very straighforward to integrate
>> - the script just need to have an option to create all reports/files in
>> the current directory rather than a temp dir:
>>
>> Infrastructure/Fedorahosted-retirement - Fedora Project Wiki
>>
>> User can initiate the report using the ABRT command line utility. Later
>> on, we can integrate Foreman to trigger ABRT report automatically on
>> exception or something like that - this is I guess out of scope of this
>> effort.
>>
>
> Keep it optional, not all our supported distros use ABRT :slight_smile:
>
>
>> Now, I am going to collect these files (skipped is not found):
>>
>
> Good list. I would add the output of "facter" to that - we often see
> machines not having the same FQDN in facter as in hostname -f.
>

YES!

>
> Keep in mind the decentralised nature of Foreman - it's quite possible
> that over half that list are from services that may be running on other
> machines through the foreman-proxy. Perhaps it could query Foreman for the
> smart proxy list and suggest further files to be collected from other
> systems?
>
> Side note: Should this be a rake task rather than a separate script?
>

This shouldn't be a rake task IMO because if a user's installation is
completely borked they should still be able to run the script so they can
get help debugging.

··· On Tue, Apr 16, 2013 at 11:34 AM, Greg Sutcliffe wrote: > On 16 April 2013 16:17, Lukas Zapletal wrote:

Greg

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

IMHO… worry about collecting the data you need to determine why the
install is busted. Other folks will integrate it into higher level tools
like abrt and sosreport.

Is there any answer file from the installer which could be grabbed?

– bk

··· On 04/16/2013 12:05 PM, Dominic Cleal wrote: > It should also be integrated into sos, since this is used for support > purposes. There's a "cloudforms" plugin today that calls katello-debug:
4

> Hello,
>
> I am starting to work on a simple katello-foreman script. It will likely
> be very similar to this one:
>
> https://github.com/Katello/katello/blob/master/src/script/katello-debug

Awesome :slight_smile:

Main purpose: collect all relevant log files, remove all possible
> passwords or secret keys, create a tarball and tell the user where to
> send/upload. The same for relevant config files.
>

Be sure to scrub sensitive data, or our users might be irritated :slight_smile:

> In the future, this can be easily integrated with ABRT, but the first
> cut is (I think) no ABRT integration. I believe the integration
> interface is ABRT Custom Hooks which is very straighforward to integrate
> - the script just need to have an option to create all reports/files in
> the current directory rather than a temp dir:
>
> https://fedorahosted.org/abrt/wiki/AbrtConfiguration
>
> User can initiate the report using the ABRT command line utility. Later
> on, we can integrate Foreman to trigger ABRT report automatically on
> exception or something like that - this is I guess out of scope of this
> effort.
>

Keep it optional, not all our supported distros use ABRT :slight_smile:

> Now, I am going to collect these files (skipped is not found):
>

Good list. I would add the output of "facter" to that - we often see
machines not having the same FQDN in facter as in hostname -f.

Keep in mind the decentralised nature of Foreman - it's quite possible that
over half that list are from services that may be running on other machines
through the foreman-proxy. Perhaps it could query Foreman for the smart
proxy list and suggest further files to be collected from other systems?

Side note: Should this be a rake task rather than a separate script?

Greg

··· On 16 April 2013 16:17, Lukas Zapletal wrote:
5

> Hello,
>
> I am starting to work on a simple katello-foreman script. It will likely
> be very similar to this one:
>
> https://github.com/Katello/katello/blob/master/src/script/katello-debug
>
> Main purpose: collect all relevant log files, remove all possible
> passwords or secret keys, create a tarball and tell the user where to
> send/upload. The same for relevant config files.
>
> In the future, this can be easily integrated with ABRT, but the first
> cut is (I think) no ABRT integration. I believe the integration
> interface is ABRT Custom Hooks which is very straighforward to integrate
> - the script just need to have an option to create all reports/files in
> the current directory rather than a temp dir:
>
> https://fedorahosted.org/abrt/wiki/AbrtConfiguration

It should also be integrated into sos, since this is used for support
purposes. There's a "cloudforms" plugin today that calls katello-debug:

https://github.com/sosreport/sosreport/blob/master/sos/plugins/cloudforms.py

ABRT also calls sosreport when a "crash" occurs (though with a limited
set of plugins enabled to reduce system load), so perhaps we should
integrate it into that somehow, rather than directly into ABRT?

> Now, I am going to collect these files (skipped is not found):
>
> - some typical information (cpu, memory, etc)
> - ip addresses, hostname
> - iptables output

Some of the generic items like iptables, CPU, libvirt and packages can
be handled by sosreport, but unfortunately that isn't shipped in Debian
at the moment (only via a PPA). I wonder if we should exclude these
items and then recommend using sosreport to gather the data instead of
us maintaining this code?

> - DNS resolve: localhost, hostname
> - content of /etc/hosts
> - foreman rails log
> - proxy main log
> - dhcp, dns, tftp log

Often syslog.

> - libvirt log
> - virsh list output (?)
> - foreman settings.yml and database.yml

  • other *.ya?ml config files, e.g. ignored_environments and email.

> - postgresql configuration
> - proxy setting.yml
> - foreman database table "settings" (?)

+1

I wonder how much other data we should capture from Foreman itself? I'd
quite like a list of registered smart proxies and their features, maybe
environments, orgs and locs too?

> - puppet main configuration
> - puppetmasterd log

  • configuration, if this is set up via Apache (e.g.
    /etc/{httpd,apache2}/conf.d/puppet.conf plus config.ru).

> - puppet logs (?)

Often syslog.

> - puppet certificates (serial ids)

There's an inventory.txt which has these.

> We have been pretty successful with katello-debug-certificates which
> prints out serial ids of all certificates and their CAs. I will
> incorporate the same behavior into foreman-debug directly.

Yes, this is needed for Puppet and Foreman certs.

> Opinions? Additional items or ideas?

I'd also add:

  • Apache configuration, including vhosts
  • package (RPM/deb) info
  • Gemfile.lock (and maybe other debug for vendor/, gem list?)
··· On 16/04/13 16:17, Lukas Zapletal wrote:

–
Dominic Cleal
Red Hat Engineering

6

>
> Hello,
>
> I am starting to work on a simple katello-foreman script. It will likely
> be very similar to this one:
>
> https://github.com/Katello/katello/blob/master/src/script/katello-debug
>
> Main purpose: collect all relevant log files, remove all possible
> passwords or secret keys, create a tarball and tell the user where to
> send/upload. The same for relevant config files.
>
> In the future, this can be easily integrated with ABRT, but the first
> cut is (I think) no ABRT integration. I believe the integration
> interface is ABRT Custom Hooks which is very straighforward to integrate
> - the script just need to have an option to create all reports/files in
> the current directory rather than a temp dir:
>
> https://fedorahosted.org/abrt/wiki/AbrtConfiguration
>
> User can initiate the report using the ABRT command line utility. Later
> on, we can integrate Foreman to trigger ABRT report automatically on
> exception or something like that - this is I guess out of scope of this
> effort.
>
> Now, I am going to collect these files (skipped is not found):
>
> - some typical information (cpu, memory, etc)
> - ip addresses, hostname
> - iptables output
>
> - DNS resolve: localhost, hostname
> - content of /etc/hosts
> - foreman rails log
> - proxy main log
> - dhcp, dns, tftp log
> - libvirt log
> - virsh list output (?)
> - foreman settings.yml and database.yml
> - postgresql configuration
> - proxy setting.yml
> - foreman database table "settings" (?)
> - puppet main configuration
> - puppetmasterd log
> - puppet logs (?)
> - puppet certificates (serial ids)
>
> (The first part can be optionally disabled - this is for the ABRT mode
> because ABRT already collects this.)
>
> The idea is you can run this on any server (e.g. DHCP server, DNS
> server) and logs are collected if the relevant files are found.
>
> We have been pretty successful with katello-debug-certificates which
> prints out serial ids of all certificates and their CAs. I will
> incorporate the same behavior into foreman-debug directly.
>
> Opinions? Additional items or ideas?
>
What about running the installer in noop mode?

Ohad
> –
> Later,
>
> Lukas "lzap" Zapletal
> irc: lzap #theforeman
>
> –
> You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.

··· On Apr 16, 2013 6:17 PM, "Lukas Zapletal" wrote: > For more options, visit https://groups.google.com/groups/opt_out. > >
7

> Be sure to scrub sensitive data, or our users might be irritated :slight_smile:

Definitely.

> Keep it optional, not all our supported distros use ABRT :slight_smile:

Sure, my ultimate plan is a bash script with few options.

> Good list. I would add the output of "facter" to that - we often see
> machines not having the same FQDN in facter as in hostname -f.

Right, will add.

> Keep in mind the decentralised nature of Foreman - it's quite possible that
> over half that list are from services that may be running on other machines
> through the foreman-proxy. Perhaps it could query Foreman for the smart
> proxy list and suggest further files to be collected from other systems?

The idea is simple - collect the resource, skip if it is not there.

Therefore I think we should deliver this script in a separate package
maybe, so users are able to install it on their proxy servers where
there is no Foreman main instance.

> Side note: Should this be a rake task rather than a separate script?

For the reason above, I would rather see no deps (bash only).

··· On Tue, Apr 16, 2013 at 04:34:04PM +0100, Greg Sutcliffe wrote:

–
Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

8

Good point, where do I find installer answers?

LZ

··· On Tue, Apr 16, 2013 at 12:08:31PM -0400, Bryan Kearney wrote: > On 04/16/2013 12:05 PM, Dominic Cleal wrote: > Is there any answer file from the installer which could be grabbed?

–
Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

9

> It should also be integrated into sos, since this is used for support
> purposes. There's a "cloudforms" plugin today that calls katello-debug:
>
> https://github.com/sosreport/sosreport/blob/master/sos/plugins/cloudforms.py
>
> ABRT also calls sosreport when a "crash" occurs (though with a limited
> set of plugins enabled to reduce system load), so perhaps we should
> integrate it into that somehow, rather than directly into ABRT?

Oh right, I have to admit I swapped ABRT with SOS. I meant SOS :wink:

> Some of the generic items like iptables, CPU, libvirt and packages can
> be handled by sosreport, but unfortunately that isn't shipped in Debian
> at the moment (only via a PPA). I wonder if we should exclude these
> items and then recommend using sosreport to gather the data instead of
> us maintaining this code?

Right, I am thinking about an option, like:

foreman-debug --no-generic --notar --dir /tmp/sos-2131223

> Often syslog.
k

> + other .ya?ml config files, e.g. ignored_environments and email.
you mean foreman/config/.ya?ml
k
> I wonder how much other data we should capture from Foreman itself? I'd
> quite like a list of registered smart proxies and their features, maybe
> environments, orgs and locs too?
good point, if there will be curl installed, I will fetch that as well
if this will be technically possible +1

I will try to gather as much as I can, but I will do tail -n5000 or
something like that for large files just not to make it ultra-big

> + configuration, if this is set up via Apache (e.g.
> /etc/{httpd,apache2}/conf.d/puppet.conf plus config.ru).
k

> There's an inventory.txt which has these.
k

> - Apache configuration, including vhosts
> - package (RPM/deb) info
> - Gemfile.lock (and maybe other debug for vendor/, gem list?)
good points okay

Uh, lot's of things to collect now! :slight_smile:

Thanks

··· On Tue, Apr 16, 2013 at 05:05:40PM +0100, Dominic Cleal wrote:

–
Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

10

What info does that give?

– bk

··· On 04/16/2013 06:15 PM, Ohad Levy wrote: > What about running the installer in noop mode? > > Ohad >
11

Good idea, I think Ohad mean something like

foreman-installer --noop

which will only try to show drift from what you have configured via
Puppet and actual state.

Please correct the command for me how to really run it.

LZ

··· On Tue, Apr 16, 2013 at 07:22:31PM -0400, Bryan Kearney wrote: > On 04/16/2013 06:15 PM, Ohad Levy wrote: > >What about running the installer in noop mode? > > > What info does that give?

–
Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

12

puppet apply --modulepath /usr/share/foreman-installer --noop

The answers file is usually
/usr/share/foreman-installer/foreman_installer/answers.yaml.

··· On 17/04/13 08:13, Lukas Zapletal wrote: > On Tue, Apr 16, 2013 at 07:22:31PM -0400, Bryan Kearney wrote: >> On 04/16/2013 06:15 PM, Ohad Levy wrote: >>> What about running the installer in noop mode? >>> >> What info does that give? > > Good idea, I think Ohad mean something like > > # foreman-installer --noop > > which will only try to show drift from what you have configured via > Puppet and actual state. > > Please correct the command for me how to really run it.

–
Dominic Cleal
Red Hat Engineering