Combined Foreman+katello test job on Jenkins

Hi all,

Eric and myself have spent some time working on ensuring that the full
Foreman test suite is run with the Katello plugin enabled, so that we
can have greater confidence of the combined result.

There's now a jenkins job[1] which is currently manual trigger only,
and currently requires Eric's PR[2] to function. The intention is to
iron out the last few wrinkles and then replace "test_katello_core"
with this job, so that Katello's test cycle includes the Foreman
tests. The items on our todo list are:

  • Fix the last 8 failures
  • Fix the hardcoded SQL migrations to allow testing on other databases

If anyone has any spare cycles, please take a look at the remaining
failures[3] and see if any of them look familiar. Most appear to be
some kind of change in expected objects, but in particular, one of the
failures is due to how Katello disables the nested behaviour of
Taxonomies, so parent_id cannot be set. This is a behaviour which is
obviously tested in Foreman, and we need to figure out how to handle
it.

I opened a PR to allow plugins to disable core tests via a custom
test_helper[4] but some people had concerns about the confusion that
might cause. It still feels like the best (where best may not equate
to good :P) solution to me, but I'm open to ideas on how to actually
resolve that failure rather than skip the test. Bear in mind the short
timescales we have before proposing complete re-implementations :stuck_out_tongue:

Comments on the above plan or patches to fix any of the tests are welcome.

TL;DR - there's a job with failures, help me fix them please :slight_smile:

Greg

[1] http://ci.theforeman.org/job/test_katello_core_gsutcliffe/
[2] https://github.com/Katello/katello/pull/4534/files
[3] http://ci.theforeman.org/job/test_katello_core_gsutcliffe/database=postgresql,ruby=1.9.3/4/testReport/
[4] https://github.com/theforeman/foreman/pull/1576

> Hi all,
>
> Eric and myself have spent some time working on ensuring that the full
> Foreman test suite is run with the Katello plugin enabled, so that we
> can have greater confidence of the combined result.
>
> There's now a jenkins job[1] which is currently manual trigger only,
> and currently requires Eric's PR[2] to function. The intention is to
> iron out the last few wrinkles and then replace "test_katello_core"
> with this job, so that Katello's test cycle includes the Foreman
> tests. The items on our todo list are:
>
> * Fix the last 8 failures
> * Fix the hardcoded SQL migrations to allow testing on other databases
>
> If anyone has any spare cycles, please take a look at the remaining
> failures[3] and see if any of them look familiar. Most appear to be
> some kind of change in expected objects, but in particular, one of the
> failures is due to how Katello disables the nested behaviour of
> Taxonomies, so parent_id cannot be set. This is a behaviour which is
> obviously tested in Foreman, and we need to figure out how to handle
> it.
>
> I opened a PR to allow plugins to disable core tests via a custom
> test_helper[4] but some people had concerns about the confusion that
> might cause. It still feels like the best (where best may not equate
> to good :P) solution to me, but I'm open to ideas on how to actually
> resolve that failure rather than skip the test. Bear in mind the short
> timescales we have before proposing complete re-implementations :stuck_out_tongue:

Having looked at this with Greg, we either need this test disabling
functionality, enable org/loc nesting again in Katello or create
settings that turn org/loc nesting on/off.

Eric

··· On Thu, Aug 7, 2014 at 6:38 AM, Greg Sutcliffe wrote:

Comments on the above plan or patches to fix any of the tests are welcome.

TL;DR - there’s a job with failures, help me fix them please :slight_smile:

Greg

[1] http://ci.theforeman.org/job/test_katello_core_gsutcliffe/
[2] https://github.com/Katello/katello/pull/4534/files
[3] http://ci.theforeman.org/job/test_katello_core_gsutcliffe/database=postgresql,ruby=1.9.3/4/testReport/
[4] https://github.com/theforeman/foreman/pull/1576


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

is org and location nesting disabled… or just org?

– bk

··· On 08/07/2014 03:14 PM, Eric D Helms wrote: > On Thu, Aug 7, 2014 at 6:38 AM, Greg Sutcliffe wrote: >> >Hi all, >> > >> >Eric and myself have spent some time working on ensuring that the full >> >Foreman test suite is run with the Katello plugin enabled, so that we >> >can have greater confidence of the combined result. >> > >> >There's now a jenkins job[1] which is currently manual trigger only, >> >and currently requires Eric's PR[2] to function. The intention is to >> >iron out the last few wrinkles and then replace "test_katello_core" >> >with this job, so that Katello's test cycle includes the Foreman >> >tests. The items on our todo list are: >> > >> >* Fix the last 8 failures >> >* Fix the hardcoded SQL migrations to allow testing on other databases >> > >> >If anyone has any spare cycles, please take a look at the remaining >> >failures[3] and see if any of them look familiar. Most appear to be >> >some kind of change in expected objects, but in particular, one of the >> >failures is due to how Katello disables the nested behaviour of >> >Taxonomies, so parent_id cannot be set. This is a behaviour which is >> >obviously tested in Foreman, and we need to figure out how to handle >> >it. >> > >> >I opened a PR to allow plugins to disable core tests via a custom >> >test_helper[4] but some people had concerns about the confusion that >> >might cause. It still feels like the best (where best may not equate >> >to good :P) solution to me, but I'm open to ideas on how to actually >> >resolve that failure rather than skip the test. Bear in mind the short >> >timescales we have before proposing complete re-implementations :P > Having looked at this with Greg, we either need this test disabling > functionality, enable org/loc nesting again in Katello or create > settings that turn org/loc nesting on/off. > > Eric >

Does that matter? It's the Org tests which are failing. The question
is, does it still make sense for Katello to be disabling org nesting?
If not, then we don't have to write a custom test_runner just to allow
a plugin to disable a core test (which has it's own implications for
plugins which might disable a test without replacing it with a new one

  • see the PR for the discussion).
··· On 7 August 2014 20:56, Bryan Kearney wrote: > is org _and_ location nesting disabled.. or just org?

Won't this make tests that disable organizations & locations fail? As far
as I know Katello does need at least one organization.

··· On Thu, Aug 7, 2014 at 9:56 PM, Bryan Kearney wrote:

On 08/07/2014 03:14 PM, Eric D Helms wrote:

On Thu, Aug 7, 2014 at 6:38 AM, Greg Sutcliffegreg.sutcliffe@gmail.com >> wrote:

Hi all,

Eric and myself have spent some time working on ensuring that the full
Foreman test suite is run with the Katello plugin enabled, so that we
can have greater confidence of the combined result.

There’s now a jenkins job[1] which is currently manual trigger only,
and currently requires Eric’s PR[2] to function. The intention is to
iron out the last few wrinkles and then replace "test_katello_core"
with this job, so that Katello’s test cycle includes the Foreman
tests. The items on our todo list are:

  • Fix the last 8 failures
  • Fix the hardcoded SQL migrations to allow testing on other databases

If anyone has any spare cycles, please take a look at the remaining
failures[3] and see if any of them look familiar. Most appear to be
some kind of change in expected objects, but in particular, one of the
failures is due to how Katello disables the nested behaviour of
Taxonomies, so parent_id cannot be set. This is a behaviour which is
obviously tested in Foreman, and we need to figure out how to handle
it.

I opened a PR to allow plugins to disable core tests via a custom
test_helper[4] but some people had concerns about the confusion that
might cause. It still feels like the best (where best may not equate
to good :P) solution to me, but I’m open to ideas on how to actually
resolve that failure rather than skip the test. Bear in mind the short
timescales we have before proposing complete re-implementations :stuck_out_tongue:

Having looked at this with Greg, we either need this test disabling
functionality, enable org/loc nesting again in Katello or create
settings that turn org/loc nesting on/off.

Eric

is org and location nesting disabled… or just org?

– bk


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Daniel Lobato

@elobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30

The Foreman core test suite assumes those are on any way - if you
disable them, lots of tests will fail. Individual tests which
temporarily alter the setting should continue to work.

··· On 8 August 2014 12:33, Daniel Lobato wrote: > Won't this make tests that disable organizations & locations fail? As far as > I know Katello does need at least one organization.

I have recently started back to working on this in earnest and while I am
close, there are a few observations that require some thought. First, the
current state of this work is captured in a new Foreman PR [1] and updates
to the Katello PR [2]. On the Foreman side, some updates to the test runner
need to be made in order for the test runner to be ran on Jenkins. The only
way I have found to do this is through custom test tasks that override the
Jenkins test runner, see [1] for more details. On the Katello side, the
original pull request [2] has been updated to reduce the number of skipped
tests (thanks in part to the removal of user attributes). If you are
curious to see a mostly green (at least from Foreman core) test run see [3].

Observations

  1. Most of the test that need to be skipped from the following categories:
    a) Disablement of organization nesting
    b) Katello hooking into host destroy actions
    c) Katello needing to hook into Organization CRUD workflow

  2. The test job currently fails on foreman_docker rubocop. Should rubocop
    rake tasks be part of the 'unit' test suite?

  3. Katello requires skipping a foreman_docker test because it is a
    dependency and enhances the 'jenkins:unit' task

The last observation is what currently has the test job stuck. Because I am
attempting to run 'jenkins:unit', which in Foreman core, runs all of the
Foreman test suites and outputs test reports for CI consumption. Typically
plugins have been built such that they enhance the base test tasks. In the
Rake world, enhancing a task means that you are adding additional tasks to
be run when that task is called. For example, when 'jenkins;unit' is run,
foreman_docker is configured to also have their unit tests and their
rubocop style checking run. This means that if another plugin includes
them, such as Katello, and wants to run the Foreman test suite to verify
those tests still pass with itself present, it will also run the
foreman_docker test suite. I believe there are pros and cons to this
approach.

Pro: ALL tests, core and plugin, that a particular plugin relies on can be
run from a single command and ensure that expectations are met.
Con: A plugin cannot reliably vet itself against the Foreman Core unit
tests only.

Solution Proposal:

Based on the above, my initial proposal would be the generation of two sets
of rake tasks. One set would allow running Foreman core unit tests only and
any enhancing of said tasks would be discouraged (ideally enforceable
through code). The second set would be a supported set of tasks designed to
be enhanced such that all plugins present are contributing their unit tests
to the same bucket to be run with a single command. Another option for the
second set would be a set of tasks that allows declaration of the set to be
run (e.g. rake plugin:tests[docker,katello,discovery].

Thoughts?

Eric

[1] https://github.com/theforeman/foreman/pull/2475
[2] https://github.com/Katello/katello/pull/4534
[3]
http://ci.theforeman.org/job/test_katello_core_gsutcliffe/14/database=postgresql,ruby=1.9.3/console

··· On Fri, Aug 8, 2014 at 7:48 AM, Greg Sutcliffe wrote:

On 7 August 2014 20:56, Bryan Kearney bryan.kearney@gmail.com wrote:

is org and location nesting disabled… or just org?

Does that matter? It’s the Org tests which are failing. The question
is, does it still make sense for Katello to be disabling org nesting?
If not, then we don’t have to write a custom test_runner just to allow
a plugin to disable a core test (which has it’s own implications for
plugins which might disable a test without replacing it with a new one

  • see the PR for the discussion).


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University