When logrotate for foreman-proxy runs and executes the configured postrotatecommand
/bin/systemctl kill --signal=SIGUSR1 foreman-proxy >/dev/null 2>&1 || true, all currently running REX jobs on that Proxy go stale. The SSH sessions are closed, and tasks stay in a running state according to Foreman. In Dynflow I can see the job as “suspended”.
This can easily be reproduced by starting a long-running rex job (like
sleep 300; echo foo), then executing the postrotate command manually.
Smart-Proxy should not drop open SSH connections when rotating logs, i.e. receiving SIGUSR1.
Foreman and Proxy versions:
Foreman and Proxy plugin versions:
- foreman-tasks 6.0.1
- foreman_expire_hosts 7.0.4
- foreman_hooks 0.3.17
- foreman_puppet 3.0.7
- foreman_remote_execution 6.0.0
- foreman_scc_manager 1.8.20
- foreman_snapshot_management 2.0.1
- foreman_templates 9.1.0
- katello 4.4.1
- puppetdb_foreman 5.0.0
Distribution and version:
Other relevant data:
This has been recently fixed in Send signal to only main process foreman-proxy on logrotate by adamruzicka · Pull Request #8835 · theforeman/foreman-packaging · GitHub which is present in 3.5+
You should be able to manually backport that change to your 3.2 installation.
Thanks for the fast reply
I will try to backport that to our installation and report back
The patch seems to solve the issue, thanks a lot @evgeni
This pull request does not solve the issue in Foreman-proxy writing into deleted log file. So to me, the whole logrotate is kind of pointless at the moment if the process keeps writing into the deleted file…
Logrotate works for me with 3.2, not sure what the problem in your environment might be.
Maybe it was introduced in 3.3, I cannot verify that, but I just checked and logrotate works as expected in our environment.
No. The problem is around for longer. Are you sure it’s working? I know it looks like it but if you check you’ll notice the logs being empty and with lsof I can see it’s using the rotate log before compression, which is why it’s deleted…
# lsof -p 168124 | grep proxy.log
smart-pro 168124 foreman-proxy 8w REG 253,0 328 21044075 /var/log/foreman-proxy/proxy.log-20221216 (deleted)
# rpm -q foreman-proxy
I’ve replied in your original topic on this, since it is better suited there