RFC: Deprecate foreman-debug in favor of sosreport


#1

About five years ago, I started writing little shell script called foreman-debug which serves as a quick tool to gather necessary data for troubleshooting. It evolved, hook mechanism was created, more and more components were added.

There is a project called SoS (or sosreport) which is an extensible, portable, support data collection tool primarily aimed at Linux distributions and other UNIX-like operating systems. Written in Python, it’s available in Fedora, CentOS/RHEL and clones, Debian, Ubuntu and other distributions.

We were aware of sosreport for some time, there is a plugin in sosreport which actually calls foreman-debug when present on the system including results of foreman-debug in a sosreport. As the sosreport utility was improving recent years and got plugins for pretty much most of the tools and components Foreman/Katello needs it’s logical step to start thinking about deprecating foreman-debug. From its extensive list, the most relevant data already collected and presented in reasonable way:

  • ansible
  • apache
  • cron
  • dhcp
  • firewalld
  • grafana
  • insights
  • ipmitool
  • logrotate
  • mongodb
  • mysql
  • named
  • openssl
  • pcp
  • postgresql
  • postfix
  • puppet
  • qpid
  • qpid_dispatch
  • salt
  • saltmaster
  • selinux
  • squid
  • ssh
  • tftpserver
  • tomcat
  • tuned
  • xinetd

Many data collected by foreman-debug is already collected by sosreport. An effort to put all the missing bits from foreman-debug into sosreport was started by Jake Hunsaker of Red Hat:

The proposal

  • Enhance sosreport to include all missing bits so reports can be used to troubleshoot Foreman, Katello (and Satellite) instances.
  • Present this as a proposal to RH support folks for feedback.
  • Present this as a proposal to our (Foreman) community for feedback.
  • Implement deprecated warning in foreman-debug - our legacy tool will operate normally but just with a warning that we are going to switch at some point.
  • Wait until foreman profile/plugin and changes ship in sosreport upstream release.
  • Remove all code from foreman-debug leaving it as a simple stub saying “This tool is deprecated, please use sosreport”

There are many advantages in using sosreport, the tool is much more flexible, configurable and better overall than our shell script called foreman-debug. There is one feature that is open question tho - foreman-debug can upload tarball via rsync to our write-only site and core members has access to it via http://debugs.theforeman.org/ but it looks like this is used approximately 5 times per month and since we now use Discourse with attachment feature, we can probably shutdown the service and ask users to upload tarballs via attachments.

Looking forward your comments and opinions.


#2

I think this sounds like a great step to unify on a single tool for handling debug reporting. Can you expand on the workflow when we need to add something to the sosreport’s debug information, and timeline to see that change propagate?


#3

Good one, this was also my initial concern. First, the workflow. Sosreport is a single package with many plugins, they all live in the main git repo on github, foreman/satellite are currently present there and the goal is to rewrite them to provide native data instead calling out foreman-debug. Changes will be done through github PR in the main repo.

Second, the timeline. There was a myth, I was explained, that sosreport release candence is tight to RHEL. It’s not true, as TurboTurtle explains in the PR: we release twice a year, roughly every six months - this is the goal, but as we are a small team with no full-time resources, and as we are often flooded with downstream requests from certain distributors around their release times, it is not something we are always able to keep to perfectly.

Then he continues: At the same time, we normally maintain master in a stable state, and new work is carried out on branches before merging, so we are usually able to issue point releases for downstreams that have particular requirements or schedules - we have done this for Fedora, Ubuntu, Debian and others in the past (including layered Red Hat products).

All and all, we should be able to send changes, expect releases twice an year which matches our upstream release cadence and have ability to do minor release when something blows up. These are rare bugs, we’ve had one or two.


#4

It is good to note that Debian and Ubuntu probably won’t rebase sosreport so if we need fixes for a release, we may miss them for a long time (probably years for Ubuntu LTS). With RHEL8 we’ll probably be in a similar situation where we still need to start.

Is there an option to ship fixes in a separate plugin to sosreport?