Foreman Core impact on plugins / CI

ohadlevy · February 12, 2014, 7:27am

Hi,

This week, a commit in foreman core, broke a plugin (katello) ability to
run its tests, while its not that common, I would like to discuss what is
the right way to handle this, the options i can see are:

test every pull request for foreman test suite + every plugin test suite.
test every foreman merged commit with every plugin.
test every plugin on their own PR/commits. (current state for selected
plugins?)

did I miss any other options? which one would you like?

Ohad

ehelms · February 12, 2014, 1:24pm

> > Hi,
> >
> > This week, a commit in foreman core, broke a plugin (katello) ability to
> > run its tests, while its not that common, I would like to discuss what
> > is the right way to handle this, the options i can see are:
> >
> > 1. test every pull request for foreman test suite + every plugin test
> suite.
>

The argument for this is obvious - plugins can be fixed so they aren't
broken by core and developers have to be aware of all plugins and work with
them or their developers. Downside is that as the number of plugins grow,
this becomes a nuisance. This could be a feature only for 'mature' or
'core-plugins'.

> > 2. test every foreman merged commit with every plugin.
>
> This is already the case for katello, other plugins could be triggered
> too. Others currently just run weekly or on their own commits.
>

The problem with this is that it doesn't really provide any benefit. Take
the Katello example,

Foreman merges code
Plugin tests are run and those plugins tests break
Plugin core test job is now broken
Developer of plugin pulls latest and works through fixing tests
Developer submits PR against their plugin
PR is not run because downstream core job (which the developer is trying
to fix) is broken and blocking

Now the developer is in a catch-22 where they have to disable their core
blocking job, re-run their PR, ignore all other PRs until that test fix is
in, merge their PR, re-enable all jobs. Rinse and repeat everytime
something breaks.

···

On Wed, Feb 12, 2014 at 4:26 AM, Dominic Cleal wrote: > On 12/02/14 07:27, Ohad Levy wrote:

test every plugin on their own PR/commits. (current state for
selected plugins?)

–
Dominic Cleal
Red Hat Engineering

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bryan_Kearney · February 12, 2014, 1:35pm

This is only an issue of the PR in 5 is against the core yes? I would
assume if it is against the plugin then it should fix itself.

– bk

···

On 02/12/2014 08:24 AM, Eric D Helms wrote: > The problem with this is that it doesn't really provide any benefit. > Take the Katello example, > > 1. Foreman merges code > 2. Plugin tests are run and those plugins tests break > 3. Plugin core test job is now broken > 4. Developer of plugin pulls latest and works through fixing tests > 5. Developer submits PR against their plugin > 6. PR is not run because downstream core job (which the developer is > trying to fix) is broken and blocking > > Now the developer is in a catch-22 where they have to disable their core > blocking job, re-run their PR, ignore all other PRs until that test fix > is in, merge their PR, re-enable all jobs. Rinse and repeat everytime > something breaks.

ehelms · February 12, 2014, 2:18pm

>
>> The problem with this is that it doesn't really provide any benefit.
>> Take the Katello example,
>>
>> 1. Foreman merges code
>> 2. Plugin tests are run and those plugins tests break
>> 3. Plugin core test job is now broken
>> 4. Developer of plugin pulls latest and works through fixing tests
>> 5. Developer submits PR against their plugin
>> 6. PR is not run because downstream core job (which the developer is
>> trying to fix) is broken and blocking
>>
>> Now the developer is in a catch-22 where they have to disable their core
>> blocking job, re-run their PR, ignore all other PRs until that test fix
>> is in, merge their PR, re-enable all jobs. Rinse and repeat everytime
>> something breaks.
>>
>
> This is only an issue of the PR in 5 is against the core yes? I would
> assume if it is against the plugin then it should fix itself.

See #6. The problem won't fix 'itself' because the PR tests will not run
because they are blocked by downstream tests that are failing.

Eric

···

On Wed, Feb 12, 2014 at 8:35 AM, Bryan Kearney wrote: > On 02/12/2014 08:24 AM, Eric D Helms wrote:

– bk

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bryan_Kearney · February 12, 2014, 2:20pm

Can you explain to me how the dependencies in the jenkins jobs are laid
out? I would assume that the plugin tests block on core tests.

– b
k

···

On 02/12/2014 09:18 AM, Eric D Helms wrote: > > > > On Wed, Feb 12, 2014 at 8:35 AM, Bryan Kearney > wrote: > > On 02/12/2014 08:24 AM, Eric D Helms wrote: > > The problem with this is that it doesn't really provide any benefit. > Take the Katello example, > > 1. Foreman merges code > 2. Plugin tests are run and those plugins tests break > 3. Plugin core test job is now broken > 4. Developer of plugin pulls latest and works through fixing tests > 5. Developer submits PR against their plugin > 6. PR is not run because downstream core job (which the developer is > trying to fix) is broken and blocking > > Now the developer is in a catch-22 where they have to disable > their core > blocking job, re-run their PR, ignore all other PRs until that > test fix > is in, merge their PR, re-enable all jobs. Rinse and repeat > everytime > something breaks. > > > This is only an issue of the PR in 5 is against the core yes? I > would assume if it is against the plugin then it should fix itself. > > > See #6. The problem won't fix 'itself' because the PR tests will not run > because they are blocked by downstream tests that are failing. > > Eric

Dominic_Cleal1 · February 12, 2014, 9:26am

> Hi,
>
> This week, a commit in foreman core, broke a plugin (katello) ability to
> run its tests, while its not that common, I would like to discuss what
> is the right way to handle this, the options i can see are:
>
> 1. test every pull request for foreman test suite + every plugin test suite.
> 2. test every foreman merged commit with every plugin.

This is already the case for katello, other plugins could be triggered
too. Others currently just run weekly or on their own commits.

···

On 12/02/14 07:27, Ohad Levy wrote:

test every plugin on their own PR/commits. (current state for
selected plugins?)

–
Dominic Cleal
Red Hat Engineering

thomasmckay · February 12, 2014, 3:00pm

> From: "Ohad Levy" <ohadlevy@gmail.com>
> To: "foreman-dev" <foreman-dev@googlegroups.com>
> Sent: Wednesday, February 12, 2014 2:27:14 AM
> Subject: [foreman-dev] Foreman Core impact on plugins / CI
>
> Hi,
>
> This week, a commit in foreman core, broke a plugin (katello) ability to
> run its tests, while its not that common, I would like to discuss what is
> the right way to handle this, the options i can see are:
>
> 1. test every pull request for foreman test suite + every plugin test suite.

If it is important that core changes do not break plugins, then run all the relevant tests.

Perhaps not all plugins are deemed worthy of full support by the foreman developers (both community and Red Hat paid). Those that are, though, really need to have their tests run and pull-requests blocked if all tests across all the supported plugins don't pass.

Specifically, katello is not an optional plugin in terms of needing to always work. It is half of a whole product that we, the Red Hat paid devs of the community, have a significant stake in. While it may be an optional install for the larger foreman community, it is not optional to run tests against in my opinion. I would suggest that there are other high value plugins that are destined to be supported in the Red Hat Satellite6 product; these too should fall into the same category: Thou shall not break.

···

----- Original Message -----

test every foreman merged commit with every plugin.

test every plugin on their own PR/commits. (current state for selected
plugins?)

did I miss any other options? which one would you like?

Ohad

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bryan_Kearney · February 12, 2014, 8:52pm

ok… so it looks like the database tests in the plugin have been red for
days… so all prs are blocked.

Tactically… whats needs to get done to open these back up?

– bk

···

On 02/12/2014 09:18 AM, Eric D Helms wrote: > See #6. The problem won't fix 'itself' because the PR tests will not run > because they are blocked by downstream tests that are failing. > > Eric

lzap · February 12, 2014, 7:39pm

> 1. test every pull request for foreman test suite + every plugin test suite.
> 2. test every foreman merged commit with every plugin.
> 3. test every plugin on their own PR/commits. (current state for selected
> plugins?)

I guess the most complex way of testing is (2) but if we decide not to
test on all ruby versions and db servers for this scenario, it could be
feasible. At least for selected plugins.

We should of course keep running unit tests for Foreman and all plugins
on their own (and on all possible ruby versions).

Now the question is how long it takes. The test_develop_pull_request
currently takes 82 minutes (10+5+5+15+8+18+14+4+4) of compute time, but
since this is run in paralel it is about 35 minutes. Assuming we will
add just 3-4 plugins and will test only against ruby 193 + postgres, we
could do this. Might need additional builder or two.

···

-- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Sam_Kottler1 · February 12, 2014, 8:55pm

>> See #6. The problem won't fix 'itself' because the PR tests will not run
>> because they are blocked by downstream tests that are failing.
>>
>> Eric
> ok… so it looks like the database tests in the plugin have been red for
> days… so all prs are blocked.
>
> Tactically… whats needs to get done to open these back up?

Either we need to revert the commit that broke Katello or manually test
the fix (is it available already?) to Katello and commit it to master
after local testing.

···

On 02/12/2014 03:52 PM, Bryan Kearney wrote: > On 02/12/2014 09:18 AM, Eric D Helms wrote:

– bk

Bryan_Kearney · February 12, 2014, 8:57pm

<DumbQuestion>
Who is working on he fix?
</DumbQuestion>

or… for you json folks

{
"DumbQuestion": "Who is working on he fix?";
}

– bk

···

On 02/12/2014 03:55 PM, Sam Kottler wrote: > > > On 02/12/2014 03:52 PM, Bryan Kearney wrote: >> On 02/12/2014 09:18 AM, Eric D Helms wrote: >>> See #6. The problem won't fix 'itself' because the PR tests will not run >>> because they are blocked by downstream tests that are failing. >>> >>> Eric >> ok.. so it looks like the database tests in the plugin have been red for >> days.. so all prs are blocked. >> >> Tactically.. whats needs to get done to open these back up? > > Either we need to revert the commit that broke Katello or manually test > the fix (is it available already?) to Katello and commit it to master > after local testing.

ehelms · February 12, 2014, 9:19pm

I have tracked it down and working through the finishing touches of the fix.

···

On Wed, Feb 12, 2014 at 3:57 PM, Bryan Kearney wrote:

On 02/12/2014 03:55 PM, Sam Kottler wrote:

On 02/12/2014 03:52 PM, Bryan Kearney wrote:

On 02/12/2014 09:18 AM, Eric D Helms wrote:

See #6. The problem won’t fix ‘itself’ because the PR tests will not run
because they are blocked by downstream tests that are failing.

Eric

ok… so it looks like the database tests in the plugin have been red for
days… so all prs are blocked.

Tactically… whats needs to get done to open these back up?

Either we need to revert the commit that broke Katello or manually test
the fix (is it available already?) to Katello and commit it to master
after local testing.

Who is working on he fix?
or… for you json folks

{
“DumbQuestion”: “Who is working on he fix?”;

}

– bk

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dLobatog · February 13, 2014, 10:51am

In general I agree with Tom McKay, Katello, Discovery, and a handful of
other plugins that are key features, or meant to be supported by Red Hat,
could be tested in the way Lukas said.
A good practice to recommend to plugin developers could be to tag their
plugins and say 'plugin XYZ version 1.0 supports Foreman up to 1.3, version
2.0 supports Foreman 1.4+, etc…'. Ideally we don't want to break
compatibility of course.

···

On Wed, Feb 12, 2014 at 10:19 PM, Eric D Helms wrote:

I have tracked it down and working through the finishing touches of the
fix.

On Wed, Feb 12, 2014 at 3:57 PM, Bryan Kearney bryan.kearney@gmail.comwrote:

On 02/12/2014 03:55 PM, Sam Kottler wrote:

On 02/12/2014 03:52 PM, Bryan Kearney wrote:

On 02/12/2014 09:18 AM, Eric D Helms wrote:

See #6. The problem won’t fix ‘itself’ because the PR tests will not
run
because they are blocked by downstream tests that are failing.

Eric

ok… so it looks like the database tests in the plugin have been red for
days… so all prs are blocked.

Tactically… whats needs to get done to open these back up?

Either we need to revert the commit that broke Katello or manually test
the fix (is it available already?) to Katello and commit it to master
after local testing.

Who is working on he fix?
or… for you json folks

{
“DumbQuestion”: “Who is working on he fix?”;

}

– bk

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
Daniel Lobato

@elobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30