Rackspace ending OSS discount program

Hi all,

As of yesterday I received a mail from Rackspace stating that they are
ending the OSS discount program at the end of Dec. I have spoken to our
Rackspace Account Manager, and there's no offer of some other kind of
discount - we'll need to migrate the workload, or pay for the usage.
Given our heavy use of Rackspace, this is going to take some work to
mitigate.

Many projects are affected by this (e.g. readthedocs[1]) and are
talking about it on Twitter - I'll be doing the same shortly. There's
already a number of companies offering to help (notably the Open Source
Labs [2]), and I'll be contacting them. However we need to decide what
to ask for.

I don't believe we'll find a single sponsor to cover the whole $2k that
we spend at Rackspace each month. Currently on Rackspace we have:

  • The core infra - puppetmaster, Jenkins master, and web node
  • The permanent Jenkins slaves - 9 of these (by far the largest cost)
  • The temporary slaves spun up by BATS and then deleted
  • The bandwidth used by the web node - Rackspace claim 6Tb / month, but
    I struggle to prove that with my own stats

I think each of these can be handled differently.

Web node / traffic - Right now we have an offer of a sponsored CDN
which I'm trying to finalize, and (separately) and offer of a
mirror+bandwidth from someone else. I'm asking them to run the web node
for us (instead of just a mirror), and with the CDN that should be
acceptable bandwidth.

Core Infra - I will ask Scaleway if we can increase our discount, and
host the Puppetmaster and Jenkins master there. We also have some
hosting with oVirt, that may be possible to increase. Failing that, OSL
or another new sponsor might be able to cover this.

Temporary slaves - if we can find someone to donate some kind of cloud
account (AWS, GCE, Digitalocean, whatever) then this ought to be
fixable in our JJB config - Eric, do I have that right?

Permanent slaves - this is the big one, we simply don't have the
compute power to replace all 9 Rackspace slaves at this time. We are
lucky in that a new slave requires no access to the rest of the infra
(inbound SSH only for the setup), so hopefully we can make a big push
for new sponsors or upgrading the ones we have.

So, I will submit the application to OSL shortly ([3] if anyone else
needs it for another project), tweet out a request for help, and I will
speak to our existing sponsors about more capacity. In the mean time,
if anyone has any contacts or offers to make, let's hear it :slight_smile:

Greg

[1] https://twitter.com/ericholscher/status/920396452307668992
[2] https://twitter.com/osuosl/status/920491352634269696
[3] http://osuosl.org/request-hosting/

Thanks for the email, Greg. It was interesting to read as it contains an
overview of our infrastructure for those of us who are less familiar. It
also kind of makes me feel like we're at a front in the quest to spread OSS.

As for constructive solutions, I have none. :slight_smile:

··· On Thu, Oct 19, 2017 at 10:03 AM, Greg Sutcliffe wrote:

Hi all,

As of yesterday I received a mail from Rackspace stating that they are
ending the OSS discount program at the end of Dec. I have spoken to our
Rackspace Account Manager, and there’s no offer of some other kind of
discount - we’ll need to migrate the workload, or pay for the usage.
Given our heavy use of Rackspace, this is going to take some work to
mitigate.

Many projects are affected by this (e.g. readthedocs[1]) and are
talking about it on Twitter - I’ll be doing the same shortly. There’s
already a number of companies offering to help (notably the Open Source
Labs [2]), and I’ll be contacting them. However we need to decide what
to ask for.

I don’t believe we’ll find a single sponsor to cover the whole $2k that
we spend at Rackspace each month. Currently on Rackspace we have:

  • The core infra - puppetmaster, Jenkins master, and web node
  • The permanent Jenkins slaves - 9 of these (by far the largest cost)
  • The temporary slaves spun up by BATS and then deleted
  • The bandwidth used by the web node - Rackspace claim 6Tb / month, but
    I struggle to prove that with my own stats

I think each of these can be handled differently.

Web node / traffic - Right now we have an offer of a sponsored CDN
which I’m trying to finalize, and (separately) and offer of a
mirror+bandwidth from someone else. I’m asking them to run the web node
for us (instead of just a mirror), and with the CDN that should be
acceptable bandwidth.

Core Infra - I will ask Scaleway if we can increase our discount, and
host the Puppetmaster and Jenkins master there. We also have some
hosting with oVirt, that may be possible to increase. Failing that, OSL
or another new sponsor might be able to cover this.

Temporary slaves - if we can find someone to donate some kind of cloud
account (AWS, GCE, Digitalocean, whatever) then this ought to be
fixable in our JJB config - Eric, do I have that right?

Permanent slaves - this is the big one, we simply don’t have the
compute power to replace all 9 Rackspace slaves at this time. We are
lucky in that a new slave requires no access to the rest of the infra
(inbound SSH only for the setup), so hopefully we can make a big push
for new sponsors or upgrading the ones we have.

So, I will submit the application to OSL shortly ([3] if anyone else
needs it for another project), tweet out a request for help, and I will
speak to our existing sponsors about more capacity. In the mean time,
if anyone has any contacts or offers to make, let’s hear it :slight_smile:

Greg

[1] https://twitter.com/ericholscher/status/920396452307668992
[2] https://twitter.com/osuosl/status/920491352634269696
[3] http://osuosl.org/request-hosting/


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Andrew Kofink
akofink@redhat.com
IRC: akofink
Associate Software Engineer
Red Hat Satellite

RedHat won't host this stuff for the project?

··· On Thu, Oct 19, 2017 at 10:03 AM, Greg Sutcliffe wrote:

Hi all,

As of yesterday I received a mail from Rackspace stating that they are
ending the OSS discount program at the end of Dec. I have spoken to our
Rackspace Account Manager, and there’s no offer of some other kind of
discount - we’ll need to migrate the workload, or pay for the usage.
Given our heavy use of Rackspace, this is going to take some work to
mitigate.

Many projects are affected by this (e.g. readthedocs[1]) and are
talking about it on Twitter - I’ll be doing the same shortly. There’s
already a number of companies offering to help (notably the Open Source
Labs [2]), and I’ll be contacting them. However we need to decide what
to ask for.

I don’t believe we’ll find a single sponsor to cover the whole $2k that
we spend at Rackspace each month. Currently on Rackspace we have:

  • The core infra - puppetmaster, Jenkins master, and web node
  • The permanent Jenkins slaves - 9 of these (by far the largest cost)
  • The temporary slaves spun up by BATS and then deleted
  • The bandwidth used by the web node - Rackspace claim 6Tb / month, but
    I struggle to prove that with my own stats

I think each of these can be handled differently.

Web node / traffic - Right now we have an offer of a sponsored CDN
which I’m trying to finalize, and (separately) and offer of a
mirror+bandwidth from someone else. I’m asking them to run the web node
for us (instead of just a mirror), and with the CDN that should be
acceptable bandwidth.

Core Infra - I will ask Scaleway if we can increase our discount, and
host the Puppetmaster and Jenkins master there. We also have some
hosting with oVirt, that may be possible to increase. Failing that, OSL
or another new sponsor might be able to cover this.

Temporary slaves - if we can find someone to donate some kind of cloud
account (AWS, GCE, Digitalocean, whatever) then this ought to be
fixable in our JJB config - Eric, do I have that right?

Permanent slaves - this is the big one, we simply don’t have the
compute power to replace all 9 Rackspace slaves at this time. We are
lucky in that a new slave requires no access to the rest of the infra
(inbound SSH only for the setup), so hopefully we can make a big push
for new sponsors or upgrading the ones we have.

So, I will submit the application to OSL shortly ([3] if anyone else
needs it for another project), tweet out a request for help, and I will
speak to our existing sponsors about more capacity. In the mean time,
if anyone has any contacts or offers to make, let’s hear it :slight_smile:

Greg

[1] https://twitter.com/ericholscher/status/920396452307668992
[2] https://twitter.com/osuosl/status/920491352634269696
[3] http://osuosl.org/request-hosting/


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

> RedHat won't host this stuff for the project?

As of today, we have some slaves hosted on the Ovirt infra, and some
applications running in a paid-for Openshift v2 account. I'm in
discussions with Ovirt about increasing that capacity, and we've
requested space on Openshift on v3. There may be other things possible,
I'm following up on that.

However the story gets yet more interesting. Today, I received this
email :slight_smile:

··· On Thu, 2017-10-19 at 12:43 -0400, Josh Baird wrote:

-------- Forwarded Message --------

Hello,
      We apologize for any confusion yesterday’s communication may

have caused. While we’re no longer accepting new applicants to the Open
Source Discount (OSS) program, all existing OSS program customers will
keep their credits. Rackspace is committed to open source technology,
and we look forward to continuing to support you and your Open Source
project.
If you have any questions, please reach out to your account
manager.

-------- End Message --------

This seems to have been sent to all OSS projects at Rackspace, see http
s://twitter.com/ericholscher/status/921109823671869440 for example.
There’s no way we were “confused” - I spoke to an Account Manager to
confirm it - so this wording seems to be a fave-saving excercise.
However, it’s a positive for us, and I appreciate their continued
support, so we won’t call them on it :slight_smile:

So, the heat is off - but for how long? Such events are usually the
start of a slow decline, and I would still be keen to move things away
from Rackspace. I think the plan now should be to move the web host,
Puppetmaster, and Jenkins master away from Rackspace, and use that
solely for CI slaves. Then if anything happens at short notice, we lose
only capacity and nothng more serious.

I’ve already had several community members reach out to me privately,
so with Rackspace staying, I’m confident we can actually increase our
slave capacity for the short term. My thanks to all of them, and stay
tuned for announcements once deals are agreed :wink:

Greg