Jenkins full disk incident

Development Infra SIG
1

Just now Jenkins had a full disk. Turns out there was a Katello PR test job build (number 9791) that had a 47GB log file. I’ve removed that log file which freed up more than enough disk space. After that I restarted the Jenkins process.

Given the excessive size of a single build (the /var/lib/jenkins partition is 100G), I’m not sure we could (or should) have planned ahead for this. I’m not sure if Jenkins has some built in limit for this to keep the size small. For now I’d suggest to keep this in mind and continue. If it pops up more often, we can look into technical solutions.

2

Thanks for info. I am also building bunch of FDI these days as I am finishing the final build. Can you guys check if this is not leaving too much behind? Each build leaves 1GB of artifacts, I really don’t need to keep more than 1-2 last artifacts. Perhaps log files longer but I am not sure if that is configurable.

https://ci.theforeman.org/job/foreman-discovery-image-release/

3

the FDI builds don’t store the ISO/TAR as artifacts, so no.