Hello,
we have merged new feature called unified and structured logging which brings journald support into Foreman core. This means that extra information is added to each log line, like request or session id, organization an location id, username or even exception class, name and backtrace. The idea is better searching, correlation and ability to integrate with ELK stack and similar tools.
The default logging setting in nightly is still file logging to /var/log/foreman/production.log, but we would like to move forward. To me, the most flexible option is to set journald logging, if operating system supports it, syslog if it does not. Let me explain why.
Journald supports structured logging as the only technology out of three, so when needed, more information can be extracted from logs, log records can be easily correlated, logs can be pulled from remote systems in a secure way (better than syslog) and there are couple of advanced things like log sealing.
Journald is installed by default on RHEL7+ and clones and it runs in transient mode, which means it maintains small memory buffer (15 MB I think) of latest log messages but does not persist them, until user tell it to do so (by creating /var/log/journal that’s all you need to do). By default, journald is configured to forward all messages to syslog, which puts them into /var/log/messages and other files according to its configuration. I believe that the similar configuration is on Debian systems at the moment.
Foreman logging into journald practically mean that logs end up in syslog where they can be left in /var/log/messages or put into different file. Foreman users will have the flexibility to keep using default setup, enable persistent journal and turn off syslog, turn off journald and only use syslog or revert to simple file logging. We also implemented JSON logging so even users who do not want to use journald can still take advantage of additional fields.
To forward all foreman logs into separate file, this rsyslog configuration file needs to be deployed which is extra work in our puppet installer (we will need to create new puppet-managed service rsyslog and syslog-ng):
local6.* /var/log/foreman/production.log
Now, we would like to ask you how you feel about this change. Let me summarize advantages of logging into system journal and/or syslog:
- structured logging by default
- sending logs over network
- no custom log file rotation needed
- advanced features (log sealing, throttling)
- easier filtering and integration with ELK
There are options:
- Let’s go for journald - Rails messages will end up in journal and/or
/var/log/messages - The same but configure syslog to put Rails logs in
/var/log/foreman/production.log - Keep using the old file appender
Let us know.