our download site is hosted on Rackspace (US) and some HTTP traffic is served by Fastly CDN (worldwide). I cannot stress enough how important our sponsors are, thank you very much!
But if you install Foreman regularly (e.g. development, testing RCs) or you want simply to support our community by providing public mirror, rsyncing our content is as easy as:
for what in yum deb downloads; do
mkdir -p $DST 2>/dev/null
rsync -rlvSHP --delete --exclude nightly --exclude scratch $SRC $DST
Remove the “–exclude nightly” if you also want nightlies which is not a bad idea at all. Please avoid mirroring around 04:00 UTC when we generate some of our nightly repos. The “scratch” directory is not needed, unless you are discovery image developer.
If you are about to set up a public mirror, reach out to us or @Gwmngilfen and we can work together putting this into http listing header text so users can decide.
This depends on our goal. If our goal is “more availability of packages” then mirrors are great. If our goal is “make sure we have a backup of files” then a single rsync mirror, over the internal Rackspace network, is probably better
I was thinking the former, although I know that long-term goal is to bring everything to Fastly and have everything served via the CDN. But even CDN won’t help if you have poor connection like me - an in-house mirror is great thing to have.
We should definitely do the latter as well, just keep in mind that rsync ... --delete is everything else but a proper backup Snapshots are also quite bad backups, we need incremental normal backups which can help us with “hey, I think I deleted a file three months ago” problems. I personally like “duplicity” as it is trivial to setup.
Edit: Public mirrors and CDN are complementary tools, if anyone is interested maintaining a mirror let’s have both.
If you scroll up the size is there, up to 60 GB including nightlies. Rsync has it’s own protocol which is often used for public content (our case), but since it is not secured, people tend to prefer SSH wrapper. But it’s not mandatory, just run my shell script and you will see.