About five years ago, I started writing little shell script called foreman-debug which serves as a quick tool to gather necessary data for troubleshooting. It evolved, hook mechanism was created, more and more components were added.
There is a project called SoS (or sosreport) which is an extensible, portable, support data collection tool primarily aimed at Linux distributions and other UNIX-like operating systems. Written in Python, it’s available in Fedora, CentOS/RHEL and clones, Debian, Ubuntu and other distributions.
We were aware of sosreport for some time, there is a plugin in sosreport which actually calls foreman-debug when present on the system including results of foreman-debug in a sosreport. As the sosreport utility was improving recent years and got plugins for pretty much most of the tools and components Foreman/Katello needs it’s logical step to start thinking about deprecating foreman-debug. From its extensive list, the most relevant data already collected and presented in reasonable way:
ansible
apache
cron
dhcp
firewalld
grafana
insights
ipmitool
logrotate
mongodb
mysql
named
openssl
pcp
postgresql
postfix
puppet
qpid
qpid_dispatch
salt
saltmaster
selinux
squid
ssh
tftpserver
tomcat
tuned
xinetd
Many data collected by foreman-debug is already collected by sosreport. An effort to put all the missing bits from foreman-debug into sosreport was started by Jake Hunsaker of Red Hat:
Enhance sosreport to include all missing bits so reports can be used to troubleshoot Foreman, Katello (and Satellite) instances.
Present this as a proposal to RH support folks for feedback.
Present this as a proposal to our (Foreman) community for feedback.
Implement deprecated warning in foreman-debug - our legacy tool will operate normally but just with a warning that we are going to switch at some point.
Wait until foreman profile/plugin and changes ship in sosreport upstream release.
Remove all code from foreman-debug leaving it as a simple stub saying “This tool is deprecated, please use sosreport”
There are many advantages in using sosreport, the tool is much more flexible, configurable and better overall than our shell script called foreman-debug. There is one feature that is open question tho - foreman-debug can upload tarball via rsync to our write-only site and core members has access to it via http://debugs.theforeman.org/ but it looks like this is used approximately 5 times per month and since we now use Discourse with attachment feature, we can probably shutdown the service and ask users to upload tarballs via attachments.
I think this sounds like a great step to unify on a single tool for handling debug reporting. Can you expand on the workflow when we need to add something to the sosreport’s debug information, and timeline to see that change propagate?
Good one, this was also my initial concern. First, the workflow. Sosreport is a single package with many plugins, they all live in the main git repo on github, foreman/satellite are currently present there and the goal is to rewrite them to provide native data instead calling out foreman-debug. Changes will be done through github PR in the main repo.
Second, the timeline. There was a myth, I was explained, that sosreport release candence is tight to RHEL. It’s not true, as TurboTurtle explains in the PR: we release twice a year, roughly every six months - this is the goal, but as we are a small team with no full-time resources, and as we are often flooded with downstream requests from certain distributors around their release times, it is not something we are always able to keep to perfectly.
Then he continues: At the same time, we normally maintain master in a stable state, and new work is carried out on branches before merging, so we are usually able to issue point releases for downstreams that have particular requirements or schedules - we have done this for Fedora, Ubuntu, Debian and others in the past (including layered Red Hat products).
All and all, we should be able to send changes, expect releases twice an year which matches our upstream release cadence and have ability to do minor release when something blows up. These are rare bugs, we’ve had one or two.
It is good to note that Debian and Ubuntu probably won’t rebase sosreport so if we need fixes for a release, we may miss them for a long time (probably years for Ubuntu LTS). With RHEL8 we’ll probably be in a similar situation where we still need to start.
Is there an option to ship fixes in a separate plugin to sosreport?
I was thinking the same, however when you miss a debug information it’s probably not big deal and you can always ask for a particular command.
I am not aware of any mechanism, however what we could do is actually making “foreman” plugin to call a shell script let’s say /usr/share/foreman/script/sosreport-extras.sh with sos report version as a parameter where we could actually provide missing bits if we run into something. This could be an elegant way of workarounding those LTS releases in case something important shows up. Opinions?
Here is my proposal to workaround slow paced sosreport rebasing in Debian OS:
Create new command “foreman-maintain sos” which creates extra debugging files if necessary (initially it will be a dummy command - does nothing)
Create a patch into sosreport to call the command above during collecting of output including all its output.
The command will accept option: --exclude feature --exclude feature2 so sosreport can actually tell which features to exclude from the foreman-maintain report because it already has it.
Everytime we need to add something important we do patch to sosreport adding it and exclude it, therefore our script only provides what’s missing from sosreport.
Currently foreman-maintain isn’t packaged for Debian so that’d need to happen first. We also have no dependency on it being installed. Should foreman-debug become a metapackage that depends on it and sosreport?
Good points. @mbacovsky or @iNecas can you comment on debian package? Are there plans? Without it my plan is useless.
Second point, it’s not metapackage yet we still need to ship it for some time until sosreport appears in the slowest distro (Debian). Then we can turn it into metapackage or better package with dummy file which will only report “use sosreport”. We should probably add “sosreport” dependency however, probably in foreman-debug subpackage.
Here is the PR which adds new command:
$ bin/foreman-maintain advanced procedure run sos --exclude a,b
================================================================================
Perform extra log collection for sosreport:
/ Performing create_directory
- Performing excluded_actions [OK]
--------------------------------------------------------------------------------
I like the idea of a foreman-debug command that is a wrapper for actual functionality for a while. Best case that’s just running sosreport, otherwise falling back to foreman-maintain and otherwise current functionality. That maintains a consistent interface for users, at least for now.
It can probably be simple bash scripts that live in the current foreman-debug repository and we select the right one for each OS in packaging.
Upstream sosreport has been already updated with all features of foreman-debug and it no longer calls it.
I am going to update foreman-debug now so it shows a WARNING: Foreman-debug will be deprecated, use sosreport instead.
It will try to find sosreport and if it finds it in particular version higher than upstream one, it will run it instead.
Foreman-maintain will be there only for edge cases when we need to add some log file to sosreport, by default it will be a NOOP (see my PR).
Once we are sure that all distros contains new-enough sosreport we can remove foreman-debug functionality and leave only the stub (warning + run sosreport).
Packaging-wise we only need to add sosreport dependency for now, correct me if I am wrong.
@lzap, I like the idea of moving the extra bits to foreman-maintain. I wondered what the output would be? Your PR in foreman-maintain gave me the answer and it makes sense.
As for Debian support, the plan is to expose foreman-maintain features when they are ready. I have ready PR with support for multiple package managers which was the biggest obstacle till now. We also have a feature that can disable certain commands and we can use it to expose just the parts supported on Debian for Debian users. We can start with version locking and sosreport and add others once ready.
There will be actually no extra bits, just a stub for now. We might add few later on. Output should be nothing, just OK, it will create a /var/tmp directory where sosreport would pick it up.
Do you have some timeframe for Debian package? If it’s “soon enough” then I am fine with pushing this forward. Thanks.
