The main reason we still use daily rsync is incremental downloads to actually help to conserve network bandwidth on your side. The only other thing is a no-delete option so when you, guys, do remove old versions, they are preserved on our side as we tend to run them longer.
@Gwmngilfen do we remove stuff these days? I can see as far back as 1.0 for YUM and 1.2 for Debian repos.
But yes, of course, if we remove things you want to preserve, having a “fresh” 1:1 mirror doesn’t help you.
If there are similar alternatives to rsync, I’d gladly switch over to them as the method does not really matter to me, only the end result - basic HTTP/HTTPS recursive download often assume re-download of everything and definitely would not work here.
I wasn’t referring to basic HTTP/S mirroring, but something along the lines of Pulp, reposync, debmirror etc. So tools that operate over HTTP/S, but understand the repository metadata and can use that to fetch the content you care about. That way we could leverage the available CDN and you’d still have a fresh mirror
Would PULP work in that sense? I was eyeing it since last year when DEB plugin was added and if can help me to accomplish the same as I can with rsync, I’d switch over to it in a flash, just don’t have much experience with Pulp yet.
I have no experience with Pulp for Debian, but for RPM you can definetly configure it to “pull in new stuff, but don’t delete anything the remote dropped”
If you have any sample configs for Pulp to set up an incremental mirror for Foreman repos that you could post on your website as a Wiki/mirroring how-to, I’m sure it would much appreciated and used by many including myself.
Just a suggestion, but we’re definitely open to other options if you had something in mind.
That’s definitely something we can arrange for!
One thing that makes me wonder, though. Our Debian repository is 54GB big, the Yum one is only 14, yet we have users that make 1-2TB traffic a month, so basically downloading the full reposet 20-ish times? I wonder if there is also something wrong with our rsyncd and it serves content as new while it isn’t?