Problem:
When logrotate for foreman-proxy runs and executes the configured postrotatecommand /bin/systemctl kill --signal=SIGUSR1 foreman-proxy >/dev/null 2>&1 || true, all currently running REX jobs on that Proxy go stale. The SSH sessions are closed, and tasks stay in a running state according to Foreman. In Dynflow I can see the job as “suspended”.
This can easily be reproduced by starting a long-running rex job (like sleep 300; echo foo), then executing the postrotate command manually. Expected outcome:
Smart-Proxy should not drop open SSH connections when rotating logs, i.e. receiving SIGUSR1.
This pull request does not solve the issue in Foreman-proxy writing into deleted log file. So to me, the whole logrotate is kind of pointless at the moment if the process keeps writing into the deleted file…
Logrotate works for me with 3.2, not sure what the problem in your environment might be.
Maybe it was introduced in 3.3, I cannot verify that, but I just checked and logrotate works as expected in our environment.
No. The problem is around for longer. Are you sure it’s working? I know it looks like it but if you check you’ll notice the logs being empty and with lsof I can see it’s using the rotate log before compression, which is why it’s deleted…