Speeding up tests

Hi,

The challenge:

As a developer, I feel under utilized waiting for tests, the current test
suite for foreman core takes over an hour, multiply it by number of really
simple code fixes based on comment and it takes up a good chunk of a day.

I would like to suggest a short and long term solution, and would
appreciate your feedback.

Short-term:

Break down test matrix into more specific groups, this will allow to
re-trigger just a smaller part of the tests vs. the entire suit today, for
example:

  • API tests
  • DB migrations
  • UI Integration tests

Long term:

Break down core into smaller repositories - This is probably a bigger topic
then just tests, but its worth raising in this context as well… the
benefits i see are:

  • smaller repos - faster tests per repo (we would still need a plugin
    matrix kind of tests done at some point)
  • better review process - smaller repos would mean the maintainers of the
    repo actually care for all/most of the pr's - today its not the case - many
    PR's are handled by sub groups (e.g. anything under webpack folder is
    "someone else")
  • potentially better API between parts - e.g. we can create a schema
    specific repo (not saying its a good idea) - which will always work with a
    dummy rails app - in the long run - this will ensure our schema is
    resilient to upgrades and model changes in core.

I would even go further and enable auto merge after review + successful
tests, e.g.

  1. PR is sent
  2. repo tests are executed and pass
  3. reviewer approve the change
  4. larger test matrix is executed and pass
  5. code get auto merged.

We should start tracking test breakages in time, if we have a test
that did not break for a year, lets delete it. As simple as that. It's
added value is almost zero.

This is not out of my head, I've heard that on a talk somewhere.
Instead of deleting, we can move it to archive - run them on a nightly
basis, if we find this too much. Frankly, I prefer deletion.

Also, let's ditch unit tests in favor of integration tests. We have a
lot of areas covered with both and this is extra work and slowing down
things.

LZ

··· On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy wrote: > Hi, > > The challenge: > > As a developer, I feel under utilized waiting for tests, the current test > suite for foreman core takes over an hour, multiply it by number of really > simple code fixes based on comment and it takes up a good chunk of a day. > > I would like to suggest a short and long term solution, and would appreciate > your feedback. > > Short-term: > > Break down test matrix into more specific groups, this will allow to > re-trigger just a smaller part of the tests vs. the entire suit today, for > example: > * API tests > * DB migrations > * UI Integration tests > * ... > > Long term: > > Break down core into smaller repositories - This is probably a bigger topic > then just tests, but its worth raising in this context as well.. the > benefits i see are: > > * smaller repos - faster tests per repo (we would still need a plugin matrix > kind of tests done at some point) > * better review process - smaller repos would mean the maintainers of the > repo actually care for all/most of the pr's - today its not the case - many > PR's are handled by sub groups (e.g. anything under webpack folder is > "someone else") > * potentially better API between parts - e.g. we can create a schema > specific repo (not saying its a good idea) - which will always work with a > dummy rails app - in the long run - this will ensure our schema is resilient > to upgrades and model changes in core. > > I would even go further and enable auto merge after review + successful > tests, e.g. > 1. PR is sent > 2. repo tests are executed and pass > 3. reviewer approve the change > 4. larger test matrix is executed and pass > 5. code get auto merged. > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

Break down test matrix into more specific groups, this will allow to
re-trigger just a smaller part of the tests vs. the entire suit today, for
example:

> * API tests
> * DB migrations
> * UI Integration tests
> * …
>
>
Probably a re-hash of what most developers know, but I'm going to do this
anyway.

Automated tests can be split into two groups – "developer" and
"everything-else" tests. The former are meant to be used often, even
continuously (and are also known as unit-tests). These tests are meant to
test individual pieces of functionality (in OOP – classes) in isolation
from the rest of the system (in other words without any dependencies,
external or other parts of the system). The rest of automated tests, such
as integration, end-to-end (including acceptance) tests can be slow and
fragile, and are oftentimes executed at some intervals (e.g. once on hour).

Of course, in rails environment we have almost unit-tests (model tests),
almost-integration tests (controller tests), and a bunch of weird (but
useful) stuff. Not sure how useful it would be to try to break up rails
tests into smaller chunks (at least as the first step) – models and
controllers are rather tightly coupled, for example. I think it would be
useful to be able to run just the rails tests however, as it will give
developers and CI an ability to quickly (or at least quicker) check if the
change breaks anything.

The rest of the tests can be executed periodically, but possibly with
different frequencies per suite of tests. One of the main and biggest
drawbacks of such an approach is that the HEAD of the development branch
can become unstable/broken. When it happens, it can be difficult to
determine what change caused the failure (I wonder if automated git-bisect
can help with this). Such an approach also requires a coordination of
efforts in locating the issue, fixing it, and in abstaining from committing
code while the build is broken .

> Long term:
>
> Break down core into smaller repositories - This is probably a bigger
> topic then just tests, but its worth raising in this context as well…
>

Can we talk about service architecture now?

-d

> We should start tracking test breakages in time, if we have a test
> that did not break for a year, lets delete it. As simple as that. It's
> added value is almost zero.
>

The only time tests should be deleted is when the functionality they verify
has been removed. Test fragility, long execution time, and complexity are
not the reasons to get rid of them, but the signs of badly factored code,
wrong level of testing, or both.

>
> This is not out of my head, I've heard that on a talk somewhere.
> Instead of deleting, we can move it to archive - run them on a nightly
> basis, if we find this too much. Frankly, I prefer deletion.
>
> Also, let's ditch unit tests in favor of integration tests. We have a
> lot of areas covered with both and this is extra work and slowing down
> things.
>
>
Let's not. Integration and higher-up tests give big gains in test coverage
quickly, but that comes at the price of fragility, slow execution, and
inability to pinpoint failures. Unit tests, in our case it will be model
and controller tests, should be the main suite of tests.

-d

··· On Tue, Nov 7, 2017 at 12:29 AM, Lukas Zapletal wrote:

LZ

On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy ohadlevy@gmail.com wrote:

Hi,

The challenge:

As a developer, I feel under utilized waiting for tests, the current test
suite for foreman core takes over an hour, multiply it by number of
really
simple code fixes based on comment and it takes up a good chunk of a day.

I would like to suggest a short and long term solution, and would
appreciate
your feedback.

Short-term:

Break down test matrix into more specific groups, this will allow to
re-trigger just a smaller part of the tests vs. the entire suit today,
for
example:

  • API tests
  • DB migrations
  • UI Integration tests

Long term:

Break down core into smaller repositories - This is probably a bigger
topic
then just tests, but its worth raising in this context as well… the
benefits i see are:

  • smaller repos - faster tests per repo (we would still need a plugin
    matrix
    kind of tests done at some point)
  • better review process - smaller repos would mean the maintainers of the
    repo actually care for all/most of the pr’s - today its not the case -
    many
    PR’s are handled by sub groups (e.g. anything under webpack folder is
    "someone else")
  • potentially better API between parts - e.g. we can create a schema
    specific repo (not saying its a good idea) - which will always work with
    a
    dummy rails app - in the long run - this will ensure our schema is
    resilient
    to upgrades and model changes in core.

I would even go further and enable auto merge after review + successful
tests, e.g.

  1. PR is sent
  2. repo tests are executed and pass
  3. reviewer approve the change
  4. larger test matrix is executed and pass
  5. code get auto merged.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

> We should start tracking test breakages in time, if we have a test
> that did not break for a year, lets delete it. As simple as that. It's
> added value is almost zero.

All tests are valuable. We do need a high test coverage in order to be
able to do things like Rails upgrades.

> This is not out of my head, I've heard that on a talk somewhere.
> Instead of deleting, we can move it to archive - run them on a nightly
> basis, if we find this too much. Frankly, I prefer deletion.

We can define these tests as a seperate set that gets only a weekly run,
but deletion is IMHO the completetly wrong way.

> Also, let's ditch unit tests in favor of integration tests. We have a
> lot of areas covered with both and this is extra work and slowing down
> things.

I think both are needed. Integration tests are much slower than unit
tests, in general.

··· On Tue, Nov 07, 2017 at 09:29:37AM +0100, Lukas Zapletal wrote: -- Michael Moll

> Hi,
>
> The challenge:
>
> As a developer, I feel under utilized waiting for tests, the current test
> suite for foreman core takes over an hour, multiply it by number of really
> simple code fixes based on comment and it takes up a good chunk of a day.

Personally, I don't feel this issue: if it's 1 hour or 5 minutes: I'm
already doing
something else in the meantime, so I would not probably see much the difference.
Locally, I usually run just the test file that touches the code I'm
working on to have
the immediate feedback on those. Waiting a bit longer on the PR is a
price I'm willing to
pay for increasing the chance that my code runs as much as possible.

>
> I would like to suggest a short and long term solution, and would appreciate
> your feedback.
>
> Short-term:
>
> Break down test matrix into more specific groups, this will allow to
> re-trigger just a smaller part of the tests vs. the entire suit today, for
> example:
> * API tests
> * DB migrations
> * UI Integration tests
> * …

Not opposed. Seems like valid request.

>
> Long term:
>
> Break down core into smaller repositories - This is probably a bigger topic
> then just tests, but its worth raising in this context as well… the
> benefits i see are:

The question: would this speed-up the things moving, in terms of getting changes
in? I would worry that the outcome would be just oposite of that.

>
> * smaller repos - faster tests per repo (we would still need a plugin matrix
> kind of tests done at some point)

We should solve the pluign matrix first, ideally on PR level: I would recommend
first mastering that before going with more split.

> * better review process - smaller repos would mean the maintainers of the
> repo actually care for all/most of the pr's - today its not the case - many
> PR's are handled by sub groups (e.g. anything under webpack folder is
> "someone else")

But still, most of the reviewers at least see, what's going on there. I know
we could achieve this with more repos as well, but frankly, not sure it would
work that much in the real life.

> * potentially better API between parts - e.g. we can create a schema
> specific repo (not saying its a good idea) - which will always work with a
> dummy rails app - in the long run - this will ensure our schema is resilient
> to upgrades and model changes in core.

I miss the implication. Could you expand on the thoughts?

>
> I would even go further and enable auto merge after review + successful
> tests, e.g.
> 1. PR is sent
> 2. repo tests are executed and pass
> 3. reviewer approve the change
> 4. larger test matrix is executed and pass
> 5. code get auto merged.

This is ortogonal to any of the previous and makes sense to me.

– Ivan

··· On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy wrote:


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

> We should start tracking test breakages in time, if we have a test
> that did not break for a year, lets delete it. As simple as that. It's
> added value is almost zero.
>
> This is not out of my head, I've heard that on a talk somewhere.
> Instead of deleting, we can move it to archive - run them on a nightly
> basis, if we find this too much. Frankly, I prefer deletion.

-1: the fact that the test doesn't fail in a year can be just due
to fact that we have not worked on that area for some time. It might
work for some project with strictly defined use-case, but for Foreman,
I don't think thats' the case.

>
> Also, let's ditch unit tests in favor of integration tests. We have a
> lot of areas covered with both and this is extra work and slowing down
> things.

+1 for integration tests > unit tests: and I'm willing to talk about deleting
unit-tests once we know that integration tests are covering the same.

I feel unit-tests more important, when working on specific algorithm etc.
We don't do it that often. We mostly integrate.

··· On Tue, Nov 7, 2017 at 9:29 AM, Lukas Zapletal wrote:

LZ

On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy ohadlevy@gmail.com wrote:

Hi,

The challenge:

As a developer, I feel under utilized waiting for tests, the current test
suite for foreman core takes over an hour, multiply it by number of really
simple code fixes based on comment and it takes up a good chunk of a day.

I would like to suggest a short and long term solution, and would appreciate
your feedback.

Short-term:

Break down test matrix into more specific groups, this will allow to
re-trigger just a smaller part of the tests vs. the entire suit today, for
example:

  • API tests
  • DB migrations
  • UI Integration tests

Long term:

Break down core into smaller repositories - This is probably a bigger topic
then just tests, but its worth raising in this context as well… the
benefits i see are:

  • smaller repos - faster tests per repo (we would still need a plugin matrix
    kind of tests done at some point)
  • better review process - smaller repos would mean the maintainers of the
    repo actually care for all/most of the pr’s - today its not the case - many
    PR’s are handled by sub groups (e.g. anything under webpack folder is
    "someone else")
  • potentially better API between parts - e.g. we can create a schema
    specific repo (not saying its a good idea) - which will always work with a
    dummy rails app - in the long run - this will ensure our schema is resilient
    to upgrades and model changes in core.

I would even go further and enable auto merge after review + successful
tests, e.g.

  1. PR is sent
  2. repo tests are executed and pass
  3. reviewer approve the change
  4. larger test matrix is executed and pass
  5. code get auto merged.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

> We can define these tests as a seperate set that gets only a weekly run,
> but deletion is IMHO the completetly wrong way.

A cleanup is reasonable, not completely wrong way. I bet that week
after we move this into 2nd tier it starts breaking and nobody will
care, because code would had been merged already. That's the point
here.

> I think both are needed. Integration tests are much slower than unit
> tests, in general.

And by the way our unit tests are not fast, really. They are slow as
hell, mainly because most of them are not really unit tests. I am
speaking to you, model tests. Let me give you an example:

test/models/architecture_test.rb

Not a unit test, not an integration test, not a system test, haven't
failed for a very long time, useless if we have the most important
stuff covered in an integration or system test, which we do.

··· -- Lukas @lzap Zapletal

So many items to unpack here that I am going to attempt to break my
response into categorizations. If I miss a key point, please point it out
so I can follow up addressing it.

Data Analysis

First, let's start with some actual data from our test runs. Let's take a
sample from a recent test run:

PR Upgrade Test: 16 minutes
PR Develop Test: 57 minutes
PR Katello Test: 40 minutes

These are run in parallel so to speed things up, we have to address the
bottleneck which is the longest running job.

For "PR Develop Test", we run the following matrix with times:

Ruby 2.2 – postgresql: 48 minutes
Ruby 2.2 – mysql: 30 minutes
Ruby 2.2 – sqlite3: 26 minutes
Ruby 2.3 – postgresql: 56 minutes
Ruby 2.4 – postgresql: 57 minutes

These, in theory, all run in parallel as well and as we can see it's the
postgresql tests that take the longest. Part of what explains this is that
we run the integration tests only against postgresql databases. If we
breakdown this job:

Tests: 2697 seconds (45 minutes) at 2.44 tests/second for 6606 tests
Setup: 12 minutes

That's 12 minutes just to install gems, install npm modules, create and
migrate the database multiple times. But, the bottleneck is still the 45
minutes it takes to run unit plus integration tests. To put things in
perspective, if you could run each of the 6606 tests in 100 ms, it would
still take 12 minutes.

Removing Unit Tests and Their Value

Whenever I question how I am thinking about testing, I usually turn to
Uncle Bob (Robert Martin) [1] and Sandi Metz [2] for enlightenment. The two
links below are just a sampling, but given their assumed authority in the
field, and their ability to make good arguments I tend to fall in line with
their thinking. I think it's important to take away from Uncle Bob what a
unit test vs. integration test is and their different values. And from
Sandi, how to think about what a unit test should be testing.

To Lukas' point, there is value in removing unit tests. But not because
they have not broken in a while. We don't merge code unless all tests are
passing. So the argument to remove tests because they have not broken does
not hold as much water given a developer has to ensure all tests are
passing. Which, in a PR based world, is the entire point of these tests. To
ensure that a developers changes do not break assumptions within the code
base. That they do not cause regressions in expected behavior. The most
user facing guardian of this is our tests. However, we can be smarter about
our tests and that's why I would encourage watching Sandi's talk, finding
other talks and focusing on making our tests better. Let's remove tests not
because they don't break, but because they aren't providing any real value.

Let's also focus on where the bottlenecks lie. If we remove 20 unit tests
because they are old and don't break, but run in 100 ms each, we've only
shaved 2 seconds off of a 45 minute test run. Jenkins output for a given PR
gives a view of what the slowest test suites are. I'd encourage developers
to start there and tackle the bottlenecks [3].

Some Solutions

I'll throw out some of my own thoughts on solutions as well as echo some
previously iterated ones.

  1. Run Test Suite Subsets Based on Changes

This is Ohad's original idea and I think it's a good one when we have
different languages and different test sets within our codebase. He and I
have chatted about this before and I am attempting to prioritize it. The
gist of the breakdown being don't run Ruby related testing if only JS
changes and vice versa. As we move towards having more of a SPA style
application, breaking the UI bits into their own repository would allow
them to iterate faster due to a faster test suite. The same might could be
said of the Ruby bits if we can find logical separation (per Ohad's other
suggestion).

  1. Re-examine Parallel of Testing

Perhaps this is already being done to the best we can, but looking again at
can we run our test suites in a more parallel fashion to breakthrough the
45 minute bottleneck and try to break it into 3 15 minutes chunks for
example.

  1. "Static" Environments

This comes from both a trend in the Jenkins world and a question from Timo
to move towards essentially Docker containers with pre-populated
environments. The idea would be to ensure these are routinely updated with
baseline dependencies installed so that setup time is reduced by having
most assets (e.g. gems and npm modules) already on the container.

  1. Quick to Fail

This is the idea that we have the test suite bail out on the first failure,
freeing up Jenkins and giving feedback sooner that the PR has issues. I am
not a huge fan of this as it has some downsides for developers who rely on
Jenkins to tell them what issues their work has versus running tests
locally first.

  1. Thoroughly Investigate Slow Tests

I mentioned this in the previous section, but dedicate some developer time
to examining why some test suites run slow and if there are things we can
change. For example, finding and suggesting tests to remove that provide no
value, looking for less resource intensive ways to do some tests, or find
flaws in how tests were crafted to begin with. An example of that last one
is a test might be seeding the database with a lot of extraneous data that
causes it to run slow due to database operations.

  1. Reduce Support Matrix

Granted, in general, if there is not significant load on Jenkins, all of
our testing runs in parallel across Jenkins executors. However, this is
generally not true for our infrastructure. Further, developers have
expressed a desire to increase the amount of testing we do by adding
plugins into the matrix.

That being said, today we support 3 databases and 3 versions of Ruby. We
attempt to give users choice in production between postgres and mysql, and
provide developers the use of a lightweight database in sqlite. Further, we
support a single RPM based production setup and multiple Debian giving us a
range of Rubies. This is compounded by wanting to test against potential
upgrades in our Rails and Ruby stack.

With choice comes burden on infrastructure and testing. I'd ask that we
consider being more opinionated and reducing this matrix. For example, if
we centralize on Forklift based development environments we could drop
sqlite. I will say up front I am less knowledgeable about Debian, but the
packaging repository makes it appear we support 4 different versions.
Perhaps we consider, if such a thing exists, locking in on LTS type
versions or dropping support sooner to focus on a few hardened environments.

Eric

[1] https://www.youtube.com/watch?v=URSWYvyc42M
[2] http://blog.cleancoder.com/uncle-bob/2017/05/05/TestDefinitions.html
[3]
http://ci.theforeman.org/job/test_develop_pr_core/14972/database=postgresql,ruby=2.3,slave=fast/testReport/(root)/

··· On Tue, Nov 7, 2017 at 5:35 AM, Lukas Zapletal wrote:

We can define these tests as a seperate set that gets only a weekly run,
but deletion is IMHO the completetly wrong way.

https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf

A cleanup is reasonable, not completely wrong way. I bet that week
after we move this into 2nd tier it starts breaking and nobody will
care, because code would had been merged already. That’s the point
here.

I think both are needed. Integration tests are much slower than unit
tests, in general.

And by the way our unit tests are not fast, really. They are slow as
hell, mainly because most of them are not really unit tests. I am
speaking to you, model tests. Let me give you an example:

test/models/architecture_test.rb

Not a unit test, not an integration test, not a system test, haven’t
failed for a very long time, useless if we have the most important
stuff covered in an integration or system test, which we do.


Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering

>So many items to unpack here that I am going to attempt to break my
>response into categorizations. If I miss a key point, please point it out
>so I can follow up addressing it.

There's a lot of good thoughts in your mail. I'm just going to jump on
one of them.

>6) Reduce Support Matrix
>
>Granted, in general, if there is not significant load on Jenkins, all of
>our testing runs in parallel across Jenkins executors. However, this is
>generally not true for our infrastructure. Further, developers have
>expressed a desire to increase the amount of testing we do by adding
>plugins into the matrix.
>
>That being said, today we support 3 databases and 3 versions of Ruby. We
>attempt to give users choice in production between postgres and mysql, and
>provide developers the use of a lightweight database in sqlite. Further, we
>support a single RPM based production setup and multiple Debian giving us a
>range of Rubies. This is compounded by wanting to test against potential
>upgrades in our Rails and Ruby stack.
>
>With choice comes burden on infrastructure and testing. I'd ask that we
>consider being more opinionated and reducing this matrix. For example, if
>we centralize on Forklift based development environments we could drop
>sqlite. I will say up front I am less knowledgeable about Debian, but the
>packaging repository makes it appear we support 4 different versions.
>Perhaps we consider, if such a thing exists, locking in on LTS type
>versions or dropping support sooner to focus on a few hardened environments.

For Debian we try to support the latest supported version and we skip
their LTS support. This allows us to to focus on usually 1 version. We
already decided to drop Jessie for 1.17 so that can be dropped from the
matrix.

For Ubuntu we currently have two LTS versions (14.04 Trusty and 16.04
Xenial) but I'm unsure about the Trusty support. I wouldn't mind
dropping it so we can focus on the latest LTS only.

If we limit to Debian Stretch and Ubuntu Xenial then we only need to
test on Ruby 2.3. For EL7 we need to choose a new version to base on.
While we could pick 2.3 so all major supported packaging is on the same
Ruby version, I think we should continue testing on Ruby 2.4 anyway.

··· On Tue, Nov 07, 2017 at 08:34:45AM -0500, Eric D Helms wrote:

Nicely written.

> Ruby 2.2 – postgresql: 48 minutes
> Ruby 2.2 – mysql: 30 minutes
> Ruby 2.2 – sqlite3: 26 minutes
> Ruby 2.3 – postgresql: 56 minutes
> Ruby 2.4 – postgresql: 57 minutes

Fun fact, the newer Ruby version the slower time.

> Setup: 12 minutes

This is one major pain we have we need to solve.

> Let's remove tests not
> because they don't break, but because they aren't providing any real value.

That's what I tried to articulate, not my idea at all. I've seen some Bob talks.

> Let's also focus on where the bottlenecks lie. If we remove 20 unit tests
> because they are old and don't break, but run in 100 ms each, we've only
> shaved 2 seconds off of a 45 minute test run. Jenkins output for a given PR
> gives a view of what the slowest test suites are. I'd encourage developers
> to start there and tackle the bottlenecks [3].

Yeah very relevant point.

> 2) Re-examine Parallel of Testing

For foreman-core tests with discovery, I am able to pull this down to
less than 4 minutes on 8 core CPU, but I see about 60 failures mostly
JavaScript. Most of these errors are:

TypeError: undefined is not an object (evaluating 'tfm.nav.init')

If you want to give this a try, it is as easy as:

be rake parallel:create
be rake parallel:migrate
be rake parallel:test

For more details visit:

> 3) "Static" Environments
>
> This comes from both a trend in the Jenkins world and a question from Timo
> to move towards essentially Docker containers with pre-populated
> environments. The idea would be to ensure these are routinely updated with
> baseline dependencies installed so that setup time is reduced by having most
> assets (e.g. gems and npm modules) already on the container.

I don't think we necessary need containers for that, I think putting
those modules and gems into git (e.g. via git annex) or to some
side-repo could do the trick. Also migrations could be solved using
similar technique.

I think introducing containers can be counter-productive. You need to
build those containers in the end, that costs time and resources.

> 4) Quick to Fail
>
> This is the idea that we have the test suite bail out on the first failure,
> freeing up Jenkins and giving feedback sooner that the PR has issues. I am
> not a huge fan of this as it has some downsides for developers who rely on
> Jenkins to tell them what issues their work has versus running tests locally
> first.

I actually like this. Although I also sometimes send small patch
without running tests and rely on Jenkins, I have no problems running
them locally when it blows up.

> 6) Reduce Support Matrix

Please let's do this. I think that maximum two Ruby versions is enough
(oldest and newest supported) and we could skip MySQL for PR tests
(but keep it in full test suite in master/develop branches).

··· -- Later, Lukas @lzap Zapletal

>> We should start tracking test breakages in time, if we have a test
>> that did not break for a year, lets delete it. As simple as that. It's
>> added value is almost zero.
>>
>> This is not out of my head, I've heard that on a talk somewhere.
>> Instead of deleting, we can move it to archive - run them on a nightly
>> basis, if we find this too much. Frankly, I prefer deletion.
>
> -1: the fact that the test doesn't fail in a year can be just due
> to fact that we have not worked on that area for some time. It might
> work for some project with strictly defined use-case, but for Foreman,
> I don't think thats' the case.
>

-1 as well for deleting the tests. They become useful once you start
refactoring things even when the code wasn't touched for a long time.

>>
>> Also, let's ditch unit tests in favor of integration tests. We have a
>> lot of areas covered with both and this is extra work and slowing down
>> things.
>
> +1 for integration tests > unit tests: and I'm willing to talk about deleting
> unit-tests once we know that integration tests are covering the same.
>
> I feel unit-tests more important, when working on specific algorithm etc.
> We don't do it that often. We mostly integrate.

+1 I see more value in integration tests in general. Unit tests for
algorithms are useful too.
I see less value in unit tests for attribute presence and simple
validations like uniqueness.

>
>>
>> LZ
>>
>>> Hi,
>>>
>>> The challenge:
>>>
>>> As a developer, I feel under utilized waiting for tests, the current test
>>> suite for foreman core takes over an hour, multiply it by number of really
>>> simple code fixes based on comment and it takes up a good chunk of a day.
>>>
>>> I would like to suggest a short and long term solution, and would appreciate
>>> your feedback.
>>>
>>> Short-term:
>>>
>>> Break down test matrix into more specific groups, this will allow to
>>> re-trigger just a smaller part of the tests vs. the entire suit today, for
>>> example:
>>> * API tests
>>> * DB migrations
>>> * UI Integration tests
>>> * …

That would definitely help to improve day-to-day developer's experience.

>>>
>>> Long term:
>>>
>>> Break down core into smaller repositories - This is probably a bigger topic
>>> then just tests, but its worth raising in this context as well… the
>>> benefits i see are:
>>>
>>> * smaller repos - faster tests per repo (we would still need a plugin matrix
>>> kind of tests done at some point)
>>> * better review process - smaller repos would mean the maintainers of the
>>> repo actually care for all/most of the pr's - today its not the case - many
>>> PR's are handled by sub groups (e.g. anything under webpack folder is
>>> "someone else")
>>> * potentially better API between parts - e.g. we can create a schema
>>> specific repo (not saying its a good idea) - which will always work with a
>>> dummy rails app - in the long run - this will ensure our schema is resilient
>>> to upgrades and model changes in core.

The biggest benefit of splitting the repos is imho the need for
stabilizing the api between components.
I believe that there would be some transfer to stability of the whole project.

··· On Tue, Nov 7, 2017 at 11:17 AM, Ivan Necas wrote: > On Tue, Nov 7, 2017 at 9:29 AM, Lukas Zapletal wrote: >> On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy wrote:

I would even go further and enable auto merge after review + successful
tests, e.g.

  1. PR is sent
  2. repo tests are executed and pass
  3. reviewer approve the change
  4. larger test matrix is executed and pass
  5. code get auto merged.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

> > Hi,
> >
> > The challenge:
> >
> > As a developer, I feel under utilized waiting for tests, the current test
> > suite for foreman core takes over an hour, multiply it by number of really
> > simple code fixes based on comment and it takes up a good chunk of a day.
>
> Personally, I don't feel this issue: if it's 1 hour or 5 minutes: I'm
> already doing
> something else in the meantime, so I would not probably see much the
> difference. Locally, I usually run just the test file that touches the code
> I'm working on to have
> the immediate feedback on those. Waiting a bit longer on the PR is a
> price I'm willing to
> pay for increasing the chance that my code runs as much as possible.

I don't think it's any issue for me either.

>
> > I would like to suggest a short and long term solution, and would
> > appreciate your feedback.
> >
> > Short-term:
> >
> > Break down test matrix into more specific groups, this will allow to
> > re-trigger just a smaller part of the tests vs. the entire suit today, for
> > example:
> > * API tests
> > * DB migrations
> > * UI Integration tests
> > * …
>
> Not opposed. Seems like valid request.

+1

> > Long term:
> >
> > Break down core into smaller repositories - This is probably a bigger
> > topic
> > then just tests, but its worth raising in this context as well… the
>
> > benefits i see are:
> The question: would this speed-up the things moving, in terms of getting
> changes in? I would worry that the outcome would be just oposite of that.
>
> > * smaller repos - faster tests per repo (we would still need a plugin
> > matrix kind of tests done at some point)
>
> We should solve the pluign matrix first, ideally on PR level: I would
> recommend first mastering that before going with more split.

I don't like the idea. My ceoncern is we'd have more repos without reviews.
>
> > * better review process - smaller repos would mean the maintainers of the
> > repo actually care for all/most of the pr's - today its not the case -
> > many
> > PR's are handled by sub groups (e.g. anything under webpack folder is
> > "someone else")
>
> But still, most of the reviewers at least see, what's going on there. I know
> we could achieve this with more repos as well, but frankly, not sure it
> would work that much in the real life.
>
> > * potentially better API between parts - e.g. we can create a schema
> > specific repo (not saying its a good idea) - which will always work with a
> > dummy rails app - in the long run - this will ensure our schema is
> > resilient to upgrades and model changes in core.
>
> I miss the implication. Could you expand on the thoughts?
>
> > I would even go further and enable auto merge after review + successful
> > tests, e.g.
> > 1. PR is sent
> > 2. repo tests are executed and pass
> > 3. reviewer approve the change
> > 4. larger test matrix is executed and pass
> > 5. code get auto merged.
>
> This is ortogonal to any of the previous and makes sense to me.

Good idea

··· On úterý 7. listopadu 2017 11:12:31 CET Ivan Necas wrote: > On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy wrote:


Marek

Another -1 for deleting based on time, +1 for revisting tests and deleting
tests with zero value (there are such tests). Honestly I think the biggest
performance improvement would be by manually going through tests and fixing
them. Minimizing DB calls where possible, remove duplicated tests etc.

··· On čtvrtek 9. listopadu 2017 10:14:33 CET Tomas Strachota wrote: > On Tue, Nov 7, 2017 at 11:17 AM, Ivan Necas wrote: > > On Tue, Nov 7, 2017 at 9:29 AM, Lukas Zapletal wrote: > >> We should start tracking test breakages in time, if we have a test > >> that did not break for a year, lets delete it. As simple as that. It's > >> added value is almost zero. > >> > >> This is not out of my head, I've heard that on a talk somewhere. > >> Instead of deleting, we can move it to archive - run them on a nightly > >> basis, if we find this too much. Frankly, I prefer deletion. > > > > -1: the fact that the test doesn't fail in a year can be just due > > to fact that we have not worked on that area for some time. It might > > work for some project with strictly defined use-case, but for Foreman, > > I don't think thats' the case. > > -1 as well for deleting the tests. They become useful once you start > refactoring things even when the code wasn't touched for a long time.


Marek

Also, let’s ditch unit tests in favor of integration tests. We have a
lot of areas covered with both and this is extra work and slowing down
things.

+1 for integration tests > unit tests: and I’m willing to talk about
deleting unit-tests once we know that integration tests are covering the
same.

I feel unit-tests more important, when working on specific algorithm etc.
We don’t do it that often. We mostly integrate.

+1 I see more value in integration tests in general. Unit tests for
algorithms are useful too.
I see less value in unit tests for attribute presence and simple
validations like uniqueness.

LZ

On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy ohadlevy@gmail.com wrote:

Hi,

The challenge:

As a developer, I feel under utilized waiting for tests, the current
test
suite for foreman core takes over an hour, multiply it by number of
really
simple code fixes based on comment and it takes up a good chunk of a
day.

I would like to suggest a short and long term solution, and would
appreciate your feedback.

Short-term:

Break down test matrix into more specific groups, this will allow to
re-trigger just a smaller part of the tests vs. the entire suit today,
for
example:

  • API tests
  • DB migrations
  • UI Integration tests

That would definitely help to improve day-to-day developer’s experience.

Long term:

Break down core into smaller repositories - This is probably a bigger
topic
then just tests, but its worth raising in this context as well… the
benefits i see are:

  • smaller repos - faster tests per repo (we would still need a plugin
    matrix kind of tests done at some point)
  • better review process - smaller repos would mean the maintainers of
    the
    repo actually care for all/most of the pr’s - today its not the case -
    many
    PR’s are handled by sub groups (e.g. anything under webpack folder is
    "someone else")
  • potentially better API between parts - e.g. we can create a schema
    specific repo (not saying its a good idea) - which will always work with
    a
    dummy rails app - in the long run - this will ensure our schema is
    resilient to upgrades and model changes in core.

The biggest benefit of splitting the repos is imho the need for
stabilizing the api between components.
I believe that there would be some transfer to stability of the whole
project.

I would even go further and enable auto merge after review + successful
tests, e.g.

  1. PR is sent
  2. repo tests are executed and pass
  3. reviewer approve the change
  4. larger test matrix is executed and pass
  5. code get auto merged.


You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,

Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group. To unsubscribe from this group and stop receiving
emails from it, send an email to
foreman-dev+unsubscribe@googlegroups.com. For more options, visit
https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group. To unsubscribe from this group and stop receiving
emails from it, send an email to
foreman-dev+unsubscribe@googlegroups.com. For more options, visit
https://groups.google.com/d/optout.