The original file /var/log/foreman-proxy/proxy.log has been rotated by logrotate, i.e. moved and then compressed. There is a logrotate configuration for the proxy:
I am absolutely sure it’s working. I have just run lsof | fgrep -e DEL -e deleted | grep pro, I get no results.
When I intentionally make foreman-proxy log an error (by querying the DHCP API for an invalid subnet), I can see the error beeing logged to /var/log/foreman-proxy/proxy.log. I can also see the logentries from yesterday’s test in the rotated file /var/log/foreman-proxy/proxy.log-20221222.gz. As already mentioned, this in 3.2:
In our EL8 test environment, I can see the same behaviour with 3.2. No open deleted files by foreman-proxy, and logs being correctly written to the new file after rotation:
Really? O.K. That is really unexpected to me. I occasionally scan for open deleted files (e.g. after updates, to see what needs to be restarted) and an open deleted log file usually means, the process didn’t get the correct signal. Obviously, then I see the exact same symptoms: process has deleted rotated logfile open and the logfile itself has size zero.
I now sent a /version to the foreman-proxy via API as that’s logged and you are correct. The message goes into proxy.log:
# cat /var/log/foreman-proxy/proxy.log
2022-12-22T10:33:58 [I] Logging file reopened via USR1 signal
2022-12-22T10:33:59 6c797e3f [I] Started GET /version
2022-12-22T10:33:59 6c797e3f [I] Finished GET /version with 200 (0.5 ms)
O.K. It works. But I find this most confusing. The symptoms I see until then cannot be distinguished to the case when it didn’t catch the signal and is actually writing into the deleted file. And the timestamp to the Logging file reopened via USR1 signal is also misleading as the signal came at a completely different time.
So my question is: isn’t it possible to close the old file on USR1? It’s pointless to keep it open and close it only on the next message written. This would at least get rid off those confusing symptoms.
Too bad. At least the logs aren’t large so it doesn’t matter so much if the open deleted file still allocates disk space until the next message is logged…