As we need to clean up some disk space on Koji, I am proposing the following changes:
Remove all locally synced Fedora repositories, and remove Fedora 27 and 28 from the external repos list in Koji. Convert Fedora 29 (only built for nightly) to point at external URL.
When Katello 4.0 is released, drop all locally synced EL 5 content.
Remove all locally synced EPEL 6 content and update external repos to point at Index of /pub/archive/epel
Convert puppetlabs external repositories to point at official Puppetlabs repositories and delete local content
Remove foreman-rails* external repositories
Remove tfm-ror51* external repository
Remove rhel7-x86_64 as we use centos7
There is likely more conversion we can do but this my proposed starting list. @evgeni@pcreech please review
Can we make 2 “When Katello 4.0 is released, drop all locally synced EL5 content”?
We need EL5 only for client-bits, and of those Katello is a heavy user, so I’d prefer not to block Katello 3.17 fixes while it is a supported target. (Quite possible this will never be needed, but if we wait, we shall wait till the correct moment).
Would replacing locally synced content with external repos mean we are saving disk space at the cost of slower builds and higher bandwidth fees (iirc koji is on aws infra)? do we have any data on how much we use these repos and how much it would cost us to do the switch? in other words, could buying bigger storage be cheaper than increasing network costs, at least for some more commonly used repos?
I believe since the connection is initiated from inside AWS reaching out there is no network cost to pay and part why @pcreech wanted to move this route. We have been using this strategy for all EL8 builds and have not noticed any issues (knock on wood). For at least Fedora and CentOS repositories, their Koji takes the same strategy (that is where we got the idea) so in theory if we are pointing to the same repositories they are pointing to we can assume the same level of stability.
We also use mrepo today which does the job but is a bit old. Perhaps if we need a cached option we could look at a tool that manages and syncs repositories. And fetches and caches it on demand.
We did see one issue with external repos: when the repo changes, but Koji didn’t regen the local metadata yet, it might end up trying to pull a package from the external one that isn’t there anymore (as those are replaced in CentOS, not added). But that can be easily fixed by manually regenerating the repo (and probably telling koji somewhere to do that itself more often). So not a big deal as such.
koji doesn’t monitor external repositories for changes by default. Administrators can enable such bejaviour with setting check_external_repos = true in kojira.conf (for details see Koji Utilities).
Will that regen all repos/tags that use the external repo? As I think all our old tags (like 1.x etc) also refer to the now external repos and regenerating those would be a waste of resources.
In the most technical sense, yes. In the practical sense, I’m curious if we’ll notice/care, especially compared with the time lost with not having an up to date repo when they happen. And if it becomes a noticiable issue, we could always up the parallel task settings on the koji-fedora28-builder or spin up a new builder for createrepo tasks (we need an el8 host for createrepo).
newRepo tasks are in general such a minor part of the load we put on koji.
This is because we can’t run el8 newRepo on the main koji. And you can’t (as far as I can tell) filter el8 vs el7 for newRepo to go to different builders.
If we’ve seen it clog before, then its probably worth evaluating if we can increase the number of slots on that builder anyway. It can probably handle more than we have on it right now. And with our increasing need for el8 builds in general it’s probably past due for us to beef up our capacity anyways.
For a test, i’ve updated the capacity on the fedora28 builder to 8 instead of 2. Looking at the cpu history over the past week, we’ve had only minor spikes in cpu usage. My assumption is we have more headroom. Since the koji server has 4 cores, and 16 capacity, i chose 8 since koji-fedora28-builder has 2 cores.
If i remember correctly, these are hooks into the CentOS mirrorlist function. The actual server hostname that could be provided could change, and if it did we would have to update everything to a different one of the URL, since koji does not have the ability to consume mirrorlists. I was not given a guarantee that this would be even remotely stable.
All tasks have been completed. We are left with just RHEL 6.6 repositories used for building EL6 client bits currently on the Koji and are down to 550GB of used space.
