Problem:
foreman-proxy keeps file descriptors of delete log files in /var/log/foreman-proxy/ . It consumes ~ 1GB per hour until disk is full. Foreman-proxy process probably creates file proxy.log.1, then gzips the file and deletes the original file BUT FD keeps open and consumes disk space
This behavior occurs randomly
Foreman and Proxy versions:
1.20.2
Other relevant data:
logs
sudo lsof +L1
ruby 62733 foreman-proxy 9u REG 252,0 2593310065 0 1313894 /var/log/foreman-proxy/proxy.log.1 (deleted)
ruby 62733 foreman-proxy 15u REG 252,0 2365372407 0 1318313 /var/log/foreman-proxy/proxy.log.1 (deleted)
ruby 62733 foreman-proxy 20u REG 252,0 2340602901 0 1318154 /var/log/foreman-proxy/proxy.log.1 (deleted)
ruby 62733 foreman-proxy 27u REG 252,0 2390022153 0 1317781 /var/log/foreman-proxy/proxy.log.1 (deleted)
ruby 62733 foreman-proxy 32u REG 252,0 2418658852 0 1314081 /var/log/foreman-proxy/proxy.log.1 (deleted)
ruby 62733 foreman-proxy 33u REG 252,0 2267874282 0 1318979 /var/log/foreman-proxy/proxy.log.1 (deleted)
ruby 62733 foreman-proxy 39u REG 252,0 730826554 0 1319069 /var/log/foreman-proxy/proxy.log.1 (deleted)
/proc/62733/fd# ls -l | grep deleted
lrwx------ 1 foreman-proxy foreman-proxy 64 Jul 4 13:19 15 -> /var/log/foreman-proxy/proxy.log.1 (deleted)
lrwx------ 1 foreman-proxy foreman-proxy 64 Jul 4 13:19 20 -> /var/log/foreman-proxy/proxy.log.1 (deleted)
lrwx------ 1 foreman-proxy foreman-proxy 64 Jul 4 13:19 27 -> /var/log/foreman-proxy/proxy.log.1 (deleted)
lrwx------ 1 foreman-proxy foreman-proxy 64 Jul 4 13:19 32 -> /var/log/foreman-proxy/proxy.log.1 (deleted)
lrwx------ 1 foreman-proxy foreman-proxy 64 Jul 4 13:19 33 -> /var/log/foreman-proxy/proxy.log.1 (deleted)
lrwx------ 1 foreman-proxy foreman-proxy 64 Jul 4 13:19 39 -> /var/log/foreman-proxy/proxy.log.1 (deleted)
lrwx------ 1 foreman-proxy foreman-proxy 64 Jul 4 13:19 9 -> /var/log/foreman-proxy/proxy.log.1 (deleted)