On July 28, 2018, at 03:02:16 UTC, koji started failing due to running out of storage on /mnt/koji
I was able to grow the filesystem to 1008G over the 886G we currently had, and Koji is currently back online as of today, July 30, 2018 at 13:17:31 UTC. This is the hard limit for the underlying type of storage we are using in koji at the moment.
I will be taking a look into the storage situation and will work to come up with a long term plan to address our ongoing storage headaches.