what you think about adding discovery to core PRs test suite, just one
Ruby version on sqlite3?
Recently we had all red situation when we bumped to Rails 5. After
Ivan fixed that, we have the same due to fact import refactoring.
These changes are good, it's just bad timing when we are doing 1.16 RC
and these unexpected regressions happen. I would like to know in
advance.
This could set a precedent, but frankly I am ok with adding more
plugins there. What's important I think is to run all core tests
without any plugin and all plugin tests (without core tests) for all
top-tier plugins. We don't do that, we are trying to achieve "ideal"
world of all combinations all rubies all db stacks individually which
is not realistic approach with our resources.
>what you think about adding discovery to core PRs test suite, just one
>Ruby version on sqlite3?
+1
>Recently we had all red situation when we bumped to Rails 5. After
>Ivan fixed that, we have the same due to fact import refactoring.
>These changes are good, it's just bad timing when we are doing 1.16 RC
>and these unexpected regressions happen. I would like to know in
>advance.
>
>This could set a precedent, but frankly I am ok with adding more
>plugins there. What's important I think is to run all core tests
>without any plugin and all plugin tests (without core tests) for all
>top-tier plugins. We don't do that, we are trying to achieve "ideal"
>world of all combinations all rubies all db stacks individually which
>is not realistic approach with our resources.
In other discussions other people have suggested that only testing with
Katello is odd and I agree. Either we test no plugins or a bunch of
popular ones. Since testing is good, I'd prefer a bunch of popular ones.
We can argue about the rules for inclusion. My suggestion is to keep it
simple: including in the main test matrix is a commitment from both
sides and we need at least 2 active developers willing to help with test
failures/potential regressions in core PRs.
···
On Wed, Sep 27, 2017 at 01:34:08PM +0200, Lukas Zapletal wrote:
Assuming that folks are working on CI redesign, can you guys include
that on your TODO please?
LZ
···
On Wed, Sep 27, 2017 at 1:46 PM, Ewoud Kohl van Wijngaarden wrote:
> On Wed, Sep 27, 2017 at 01:34:08PM +0200, Lukas Zapletal wrote:
>>
>> what you think about adding discovery to core PRs test suite, just one
>> Ruby version on sqlite3?
>
>
> +1
>
>> Recently we had all red situation when we bumped to Rails 5. After
>> Ivan fixed that, we have the same due to fact import refactoring.
>> These changes are good, it's just bad timing when we are doing 1.16 RC
>> and these unexpected regressions happen. I would like to know in
>> advance.
>>
>> This could set a precedent, but frankly I am ok with adding more
>> plugins there. What's important I think is to run all core tests
>> without any plugin and all plugin tests (without core tests) for all
>> top-tier plugins. We don't do that, we are trying to achieve "ideal"
>> world of all combinations all rubies all db stacks individually which
>> is not realistic approach with our resources.
>
>
> In other discussions other people have suggested that only testing with
> Katello is odd and I agree. Either we test no plugins or a bunch of popular
> ones. Since testing is good, I'd prefer a bunch of popular ones.
>
> We can argue about the rules for inclusion. My suggestion is to keep it
> simple: including in the main test matrix is a commitment from both sides
> and we need at least 2 active developers willing to help with test
> failures/potential regressions in core PRs.
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> > what you think about adding discovery to core PRs test suite, just
> > one Ruby version on sqlite3?
>
> +1
This would be PR-blocking, I assume? I'm fine with that, just checking.
>
> In other discussions other people have suggested that only testing
> with Katello is odd and I agree. Either we test no plugins or a bunch
> of popular ones. Since testing is good, I'd prefer a bunch of popular
> ones.
Overall, I agree. However …
The primary concern I recall there was about load on Jenkins. We
already exceed our free allowance from Rackspace every single month
(although not by much), so more VMs / compute time there will be hard.
Do we have any suggestions for lowering the current usage so that
there is capacity for this? Some thoughts off the top of my head:
Bandwidth takes about 1/3rd of our $2k budget - perhaps we can affect
that
Take another look at running some loads on the CentOS Ci project
instead of our own
Find some new slaves from somewhere
> We can argue about the rules for inclusion. My suggestion is to keep
> it simple: including in the main test matrix is a commitment from
> both sides and we need at least 2 active developers willing to help
> with test failures/potential regressions in core PRs.
I agree, it'll need a commitment from the plugin side to deal with
broken tests. We should also have a contingency in case that's not met
within a given timeframe (as in, someone needs to acknowledge and start
working on the issue within X time) - probably removing the plugin from
the blocking job so that core is unblocked.
Greg
···
On Wed, 2017-09-27 at 13:46 +0200, Ewoud Kohl van Wijngaarden wrote:
> On Wed, Sep 27, 2017 at 01:34:08PM +0200, Lukas Zapletal wrote:
> > > what you think about adding discovery to core PRs test suite, just
> > > one Ruby version on sqlite3?
> >
>
> This would be PR-blocking, I assume? I'm fine with that, just checking.
While I'm +1 about this in general, the two specific cases for Discovery
the last weeks were Rails 4.2 -> 5.0 and separation of puppet facts
from core. I can't see how a merge block on plugin breakages can be
mandatory. Of course it's nice to know that a change is breaking
plugins, and ideally the core PR author provides PRs to the affected
plugins then, but I worry about further reduction of core
development/merge speed, which is already not at its best…
> > In other discussions other people have suggested that only testing
> > with Katello is odd and I agree. Either we test no plugins or a bunch
> > of popular ones. Since testing is good, I'd prefer a bunch of popular
> > ones.
>
> Overall, I agree. However …
>
> The primary concern I recall there was about load on Jenkins. We
> already exceed our free allowance from Rackspace every single month
> (although not by much), so more VMs / compute time there will be hard.
>
> Do we have any suggestions for lowering the current usage so that
> there is capacity for this? Some thoughts off the top of my head:
>
> * Bandwidth takes about 1/3rd of our $2k budget - perhaps we can affect
> that
Can we somehow see which part is taking most? I.e. is it the CI part or
rather the website?
Regards
···
On Wed, Sep 27, 2017 at 04:18:16PM +0100, Greg Sutcliffe wrote:
> > On Wed, Sep 27, 2017 at 01:34:08PM +0200, Lukas Zapletal wrote:
--
Michael Moll
>
> > This would be PR-blocking, I assume? I'm fine with that, just
> > checking.
>
> While I'm +1 about this in general, the two specific cases for
> Discovery
> the last weeks were Rails 4.2 -> 5.0 and separation of puppet facts
> from core. I can't see how a merge block on plugin breakages can be
> mandatory. Of course it's nice to know that a change is breaking
> plugins, and ideally the core PR author provides PRs to the affected
> plugins then, but I worry about further reduction of core
> development/merge speed, which is already not at its best…
Hmm, good point. There will always be a mix of cases where core is at
fault, or where it's just a necessary update-and-roll-forward. I think
there's two views here, we could go either way:
This is advisory-only (non-blocking), but has the advantage of
making more people aware of potential plugin breakage in PRs (compared
to only the authors really finding out in a few days when the
standalone plugin tests get run)
or
This is blocking, but the processes that were discussed a few weeks
back, about when to merge things with red tests, needs updating to
account for this new test matrix.
> > * Bandwidth takes about 1/3rd of our $2k budget - perhaps we can
> > affect that
>
> Can we somehow see which part is taking most? I.e. is it the CI part
> or rather the website?
I'll see what I can do. The Rackspace billing does provide an extensive
CSV of all charges (several MB, which for CSV is impressive :P), so I
may well be able to pull something from that.
Greg
···
On Wed, 2017-09-27 at 17:32 +0200, Michael Moll wrote:
… have you considered using some kind of a CDN for downloads, e.g. cloudflare, if traffic is a concern?
Timo
···
> On 27. Sep 2017, at 17:42, Greg Sutcliffe wrote:
>
>> On Wed, 2017-09-27 at 17:32 +0200, Michael Moll wrote:
>>
>>> This would be PR-blocking, I assume? I'm fine with that, just
>>> checking.
>>
>> While I'm +1 about this in general, the two specific cases for
>> Discovery
>> the last weeks were Rails 4.2 -> 5.0 and separation of puppet facts
>> from core. I can't see how a merge block on plugin breakages can be
>> mandatory. Of course it's nice to know that a change is breaking
>> plugins, and ideally the core PR author provides PRs to the affected
>> plugins then, but I worry about further reduction of core
>> development/merge speed, which is already not at its best...
>
> Hmm, good point. There will always be a mix of cases where core is at
> fault, or where it's just a necessary update-and-roll-forward. I think
> there's two views here, we could go either way:
>
> 1) This is advisory-only (non-blocking), but has the advantage of
> making more people aware of potential plugin breakage in PRs (compared
> to only the authors really finding out in a few days when the
> standalone plugin tests get run)
>
> or
>
> 2) This is blocking, but the processes that were discussed a few weeks
> back, about when to merge things with red tests, needs updating to
> account for this new test matrix.
>
>>> * Bandwidth takes about 1/3rd of our $2k budget - perhaps we can
>>> affect that
>>
>> Can we somehow see which part is taking most? I.e. is it the CI part
>> or rather the website?
>
> I'll see what I can do. The Rackspace billing does provide an extensive
> CSV of all charges (several MB, which for CSV is impressive :P), so I
> may well be able to pull something from that.
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Total bandwidth charge last month: $723.15
Bandwidth charges for web02: $706.79
What's harder to tell is who's consuming it - i.e. is it our users, or
just the repeated runs of the installer etc in BATS?
In any case, yes, some kind of offloading of this would give us vastly
more capability in the CI area.
Greg
···
On Wed, 2017-09-27 at 19:26 +0200, Timo Goebel wrote:
> ... have you considered using some kind of a CDN for downloads, e.g.
> cloudflare, if traffic is a concern?
>
> > I'll see what I can do. The Rackspace billing does provide an
> > extensive CSV of all charges (several MB, which for CSV is
> > impressive :P), so I may well be able to pull something from that.
fastly.com is usually happy to sponsor bandwidth for FOSS projects, if
you ask nicely (they sponsor one half of deb.debian.org, for example,
the other half is sponsored by AMZN).
I could talk to the relevant people if that is something we want.
Evgeni Golov
Software Engineer
···
On Wed, Sep 27, 2017 at 7:26 PM, Timo Goebel wrote:
> ... have you considered using some kind of a CDN for downloads, e.g. cloudflare, if traffic is a concern?
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O’Neill, Eric Shander
I did help for a bit with oVirt and with yum you can relatively easily
put up mirrors with a mirrorlist and have it automatically select the
best one.
How much disk space is needed to mirror the yum repo? I know SNT1 is
limited on that but not bandwidth since they're on a university 10Gb/s
link. If you have a mirrorlist file per release then you allow for just
mirroring the last n releases, saving a fair amount of it.
Many users are also doing some hosting so I wouldn't be surprised if you
could easily find a few to sponsor a mirror.
···
On Thu, Sep 28, 2017 at 10:07:45AM +0100, Greg Sutcliffe wrote:
>On Wed, 2017-09-27 at 19:26 +0200, Timo Goebel wrote:
>> ... have you considered using some kind of a CDN for downloads, e.g.
>> cloudflare, if traffic is a concern?
>>
>> > I'll see what I can do. The Rackspace billing does provide an
>> > extensive CSV of all charges (several MB, which for CSV is
>> > impressive :P), so I may well be able to pull something from that.
>
>So yes, it's our package mirror:
>
>Total bandwidth charge last month: $723.15
>Bandwidth charges for web02: $706.79
>
>What's harder to tell is who's consuming it - i.e. is it our users, or
>just the repeated runs of the installer etc in BATS?
>
>In any case, yes, some kind of offloading of this would give us vastly
>more capability in the CI area.
THis has gone rather quiet here, but I've not heard any opposition to
trying a CDN for the package downloads. Assuming no one has an
objection, we can start looking at this shortly.
Greg
···
On Thu, 2017-09-28 at 16:51 +0200, Evgeni Golov wrote:
> On Wed, Sep 27, 2017 at 7:26 PM, Timo Goebel > wrote:
> > ... have you considered using some kind of a CDN for downloads,
> > e.g. cloudflare, if traffic is a concern?
>
> fastly.com is usually happy to sponsor bandwidth for FOSS projects,
> if you ask nicely (they sponsor one half of deb.debian.org, for
> example, the other half is sponsored by AMZN).
>
> I could talk to the relevant people if that is something we want.
>
> I did help for a bit with oVirt and with yum you can relatively
> easily put up mirrors with a mirrorlist and have it automatically
> select the best one.
>
> How much disk space is needed to mirror the yum repo? I know SNT[1]
> is limited on that but not bandwidth since they're on a university
> 10Gb/s link. If you have a mirrorlist file per release then you allow
> for just mirroring the last n releases, saving a fair amount of it.
du -sh yum/htdocs/
9.8G yum/htdocs/
du -sh deb/htdocs/
37G deb/htdocs/
> Many users are also doing some hosting so I wouldn't be surprised if
> you could easily find a few to sponsor a mirror.
>
> [1]: http://ftp.snt.utwente.nl/
I assume building a mirroring infrastructure isn't especially hard?
I can ask on the users list / twitter etc and see if we get any
volunteers, if people think that's a way forward. I agree, I'm sure we
can find a few.
Greg
···
On Thu, 2017-09-28 at 11:21 +0200, Ewoud Kohl van Wijngaarden wrote:
Generally like self-maintained infra but there are some big benefits of
using a CDN. For us it's simple and shouldn't require too much effort
and users get lower latencies.
Let's try it. Luckily we already split the server hostname from the
service hostname so eventually we can easily migrate that without
requiring all users to change their configs.
···
On Wed, Oct 04, 2017 at 11:13:07AM +0100, Greg Sutcliffe wrote:
>On Thu, 2017-09-28 at 16:51 +0200, Evgeni Golov wrote:
>> On Wed, Sep 27, 2017 at 7:26 PM, Timo Goebel >> wrote:
>> > ... have you considered using some kind of a CDN for downloads,
>> > e.g. cloudflare, if traffic is a concern?
>>
>> fastly.com is usually happy to sponsor bandwidth for FOSS projects,
>> if you ask nicely (they sponsor one half of deb.debian.org, for
>> example, the other half is sponsored by AMZN).
>>
>> I could talk to the relevant people if that is something we want.
>
>THis has gone rather quiet here, but I've not heard any opposition to
>trying a CDN for the package downloads. Assuming no one has an
>objection, we can start looking at this shortly.
