Default logging in foreman: file, syslog or journald?


#1

Hello,

we have merged new feature called unified and structured logging which brings journald support into Foreman core. This means that extra information is added to each log line, like request or session id, organization an location id, username or even exception class, name and backtrace. The idea is better searching, correlation and ability to integrate with ELK stack and similar tools.

The default logging setting in nightly is still file logging to /var/log/foreman/production.log, but we would like to move forward. To me, the most flexible option is to set journald logging, if operating system supports it, syslog if it does not. Let me explain why.

Journald supports structured logging as the only technology out of three, so when needed, more information can be extracted from logs, log records can be easily correlated, logs can be pulled from remote systems in a secure way (better than syslog) and there are couple of advanced things like log sealing.

Journald is installed by default on RHEL7+ and clones and it runs in transient mode, which means it maintains small memory buffer (15 MB I think) of latest log messages but does not persist them, until user tell it to do so (by creating /var/log/journal that’s all you need to do). By default, journald is configured to forward all messages to syslog, which puts them into /var/log/messages and other files according to its configuration. I believe that the similar configuration is on Debian systems at the moment.

Foreman logging into journald practically mean that logs end up in syslog where they can be left in /var/log/messages or put into different file. Foreman users will have the flexibility to keep using default setup, enable persistent journal and turn off syslog, turn off journald and only use syslog or revert to simple file logging. We also implemented JSON logging so even users who do not want to use journald can still take advantage of additional fields.

To forward all foreman logs into separate file, this rsyslog configuration file needs to be deployed which is extra work in our puppet installer (we will need to create new puppet-managed service rsyslog and syslog-ng):

local6.* /var/log/foreman/production.log

Now, we would like to ask you how you feel about this change. Let me summarize advantages of logging into system journal and/or syslog:

  • structured logging by default
  • sending logs over network
  • no custom log file rotation needed
  • advanced features (log sealing, throttling)
  • easier filtering and integration with ELK

There are options:

  • Let’s go for journald - Rails messages will end up in journal and/or /var/log/messages
  • The same but configure syslog to put Rails logs in /var/log/foreman/production.log
  • Keep using the old file appender

Let us know.


#2

Thanks for the detailed overview! I created a poll for you:

  • Let’s go for journald - Rails messages will end up in journal and/or /var/log/messages
  • The same but configure syslog to put Rails logs in /var/log/foreman/production.log
  • Keep using the old file appender

0 voters


#3

Thanks, I think I did not mention that configuring syslog means that we need to start managing the service since our installer is puppet-based which is something that we want to avoid long term. Red Hat only ships rsyslog but in Debian there is also syslog-ng with different configuration, which is quite of Puppet coding because of a simple line.

I wish this could be added as unmanaged service somehow, but I am not Puppet expert. If you have some ideas, throw them in.


#4

I don’t think this is a puppet limitation, rather philosophical question. Should we touch services partly, without taking over their configuration fully? I think in devops world, the answer is either fully managed or do not touch at all. Touching one config file of rsyslog/syslog-ng might clash with other modules that configure the system.

I’d challenge this a bit since the major use of our puppet modules is installation. Using them separately is nice benefit, but should not prevent improving our installer. If adding it to puppet-foreman is not acceptable, I think we could add puppet-foreman-logging as a separate module or installer hook, both seems hackish to me. Other option I see is an if condition for whether it should drop the config file for rsyslog/syslog-ng, puppet-foreman would default to false but our installer answer file could be set to true.


#5

Another thing to note is that foreman-journald on Debian always needs to compile some native extensions on installation. I don’t know our Debian packaging, but it feels like we should be able to ship some binaries.


#6

It’s not just foreman-journald but about a dozen of our native dependencies. For Debian, we simply “gem install” all of them if I am not mistaken.


#7

FYI here is WIP branch of structured logging documentation: