Logging to stdout/journal by default

aruzicka · January 6, 2023, 11:40am

The proposal

Foreman has been logging to /var/log/foreman/production.log (and smart proxy to /var/log/foreman-proxy/proxy.log) for as long as I can remember. In the mean time, the world has moved a bit, all officially supported platforms(*1) now have systemd, containers are on the rise and one could say logging into files is becoming a thing of the past. My proposal would be to move away from it where possible and either log to journal directly or just log to stdout and then have systemd (or the container runtime) collect it. As far as I know all the bits and pieces are already there, we would “just” need to change the default configuration.

This proposal shares some points with https://community.theforeman.org/t/default-logging-in-foreman-file-syslog-or-journald but focuses on journald or stdout instead of journald or syslog.

Benefits of logging to stdout + journal

We would not have to care about logrotate. It would also completely prevent a certain class of bugs from happening (processes logging into rotated files).

Currently, both Foreman (the web application), dynflow-sidekiq workers and various rake tasks triggered from cron or manually log into the same file without any indication, making it hard to guess where parts of logs come from. This could be taken even further by turning our cron jobs into systemd timers, but that’s a topic for another RFC.

One would be able to collect logs with timestamps recalculated to an arbitrary timezone.

Once we move away from logging to files, we’d be just a step away from doing structured logging (available since Foreman 1.18) as described in the post linked above.

Disadvantages

While the change to make it happen could be somewhat small, it would additionally have to be reflected in the docs, debugging and support tools would have to be updated and it would probably take some time until people would get used to the new way of interacting with logs.

We may need start putting some journald configuration drop-ins in place to ensure logs are persisted for long enough.

*1 Yes, I know there are packages for FreeBSD and deb packages could be possibly used on devuan but it is not officially supported

ekohl · January 6, 2023, 12:02pm

I’m in favor of defaulting to journald. We already only provide systemd service files so anyone who doesn’t use systemd has to manually configure it. It’s also not like we would drop the code to log to files (right?) so users have a fallback.

Today we have foreman-rake errors:fetch_log request_id=$ID which only works on file logging:

github.com

theforeman/foreman/blob/d153a5987c50754a2b976f4ab433e44944fb47a1/lib/tasks/errors.rake#L23-L28


      
          type = Foreman::Logging.config[:type]
          unless type == 'file'
            puts "This rake task only support log file, you're using #{type}"
            puts "You can search the logs in #{type} for request_id #{request_id}"
            exit(1)
          end

It’s probably not hard to rewrite this to retrieve it from the journal, but I’d consider it a blocker for switching the default since it’s what we recommend users.

aruzicka · January 6, 2023, 3:53pm

Right, I didn’t mention this explicitly, but I’m not saying we should drop it from the codebase completely. Just set the default to something else.

Not hard if we don’t mind foreman user being in the systemd-journal group Fixes #35903 - Support extracting logs from journald by adamruzicka · Pull Request #9576 · theforeman/foreman · GitHub . If we do mind it, then things could become a bit hairy