as I was implementing dynflow smart proxy core structured logging I noticed that foreman proxy now has an improved logging in Foreman 1.20+ which by default performs auto log file rotation (100 MiB or week old what comes first). We still ship logrotate script which rotates files daily. We should pick one or another otherwise rotated logs might interfere, thus starting this thread.
I propose for foreman-proxy and foreman-proxy dynflow core to let the logging stack to rotate the files and disable (remove) logrotate script for these. For foreman core we stick with logrotate as the implementation there does not rotate files at all.
No one interested? Foreman, Proxy and Dynflow now supports self-rotating and it’s actually set. I would like to consolidate this. Here are pros and cons:
Logrotate pros:
Can compress old files
Works fine for decades
Logrotate cons:
We need to maintain extra scripts
Need to maintain implementation of signal handling or/and copy-truncate (which we use for Foreman today)
Copy-truncate can possibly lost logs while rotating (Foreman core only)
SELinux policies
Self-rotating pros:
Works out of box, no logrotate needed
Never lost a log record
Self-rotating cons:
Compression of rotated logs not possible
Here is how self-rotating setup could look like:
Remove foreman, smart proxy and dynflow logrorate scripts
Configure log rotation by logging gem to do self rotation:
Maximum log size 200 MiB
Maximum log age 1 day
Keeping last 9 files (so we avoid 1 vs 10 ordering issues)
Naming pattern: file.1.log, file.2.log
This gives maximum size occupied by logs at 2 GiB
While I like old rotated logs compressed, let me think about this for a while. Disk space is cheap, compressing logs burns some CPU time (and servers do this every day, 365 days per year) and handling of compressed logs is usually more work, in the end people tend to re-compress them. So not compressing is not totally bad idea.
Another alternative would be just rely on journald from now on where rotation is not an issue. Or keeping implementation of logrotate. Looking in the logging gem roll implementation, combination of logrotate and self-logging is also possible if logrotate can be configured only to compress and not actually perform rotation.
During debugging of file rolling (self-rotating logs) we’ve stumbled upon a bug in the logging gem. In multithreaded environment, it does not work correctly. I fixed but it might take some time to get it.
For this reason I will go ahead and turn off this in smart proxy and dynflow, the capability will be still present and if user wants she can configure this via settings but our deployments will continue to rely on logrotate with SIGUSR1 for reopening files.
