Eric and myself have spent some time working on ensuring that the full
Foreman test suite is run with the Katello plugin enabled, so that we
can have greater confidence of the combined result.
There's now a jenkins job[1] which is currently manual trigger only,
and currently requires Eric's PR[2] to function. The intention is to
iron out the last few wrinkles and then replace "test_katello_core"
with this job, so that Katello's test cycle includes the Foreman
tests. The items on our todo list are:
Fix the last 8 failures
Fix the hardcoded SQL migrations to allow testing on other databases
If anyone has any spare cycles, please take a look at the remaining
failures[3] and see if any of them look familiar. Most appear to be
some kind of change in expected objects, but in particular, one of the
failures is due to how Katello disables the nested behaviour of
Taxonomies, so parent_id cannot be set. This is a behaviour which is
obviously tested in Foreman, and we need to figure out how to handle
it.
I opened a PR to allow plugins to disable core tests via a custom
test_helper[4] but some people had concerns about the confusion that
might cause. It still feels like the best (where best may not equate
to good :P) solution to me, but I'm open to ideas on how to actually
resolve that failure rather than skip the test. Bear in mind the short
timescales we have before proposing complete re-implementations
Comments on the above plan or patches to fix any of the tests are welcome.
TL;DR - there's a job with failures, help me fix them please
> Hi all,
>
> Eric and myself have spent some time working on ensuring that the full
> Foreman test suite is run with the Katello plugin enabled, so that we
> can have greater confidence of the combined result.
>
> There's now a jenkins job[1] which is currently manual trigger only,
> and currently requires Eric's PR[2] to function. The intention is to
> iron out the last few wrinkles and then replace "test_katello_core"
> with this job, so that Katello's test cycle includes the Foreman
> tests. The items on our todo list are:
>
> * Fix the last 8 failures
> * Fix the hardcoded SQL migrations to allow testing on other databases
>
> If anyone has any spare cycles, please take a look at the remaining
> failures[3] and see if any of them look familiar. Most appear to be
> some kind of change in expected objects, but in particular, one of the
> failures is due to how Katello disables the nested behaviour of
> Taxonomies, so parent_id cannot be set. This is a behaviour which is
> obviously tested in Foreman, and we need to figure out how to handle
> it.
>
> I opened a PR to allow plugins to disable core tests via a custom
> test_helper[4] but some people had concerns about the confusion that
> might cause. It still feels like the best (where best may not equate
> to good :P) solution to me, but I'm open to ideas on how to actually
> resolve that failure rather than skip the test. Bear in mind the short
> timescales we have before proposing complete re-implementations
Having looked at this with Greg, we either need this test disabling
functionality, enable org/loc nesting again in Katello or create
settings that turn org/loc nesting on/off.
Eric
···
On Thu, Aug 7, 2014 at 6:38 AM, Greg Sutcliffe wrote:
Comments on the above plan or patches to fix any of the tests are welcome.
TL;DR - there’s a job with failures, help me fix them please
is org and location nesting disabled… or just org?
– bk
···
On 08/07/2014 03:14 PM, Eric D Helms wrote:
> On Thu, Aug 7, 2014 at 6:38 AM, Greg Sutcliffe wrote:
>> >Hi all,
>> >
>> >Eric and myself have spent some time working on ensuring that the full
>> >Foreman test suite is run with the Katello plugin enabled, so that we
>> >can have greater confidence of the combined result.
>> >
>> >There's now a jenkins job[1] which is currently manual trigger only,
>> >and currently requires Eric's PR[2] to function. The intention is to
>> >iron out the last few wrinkles and then replace "test_katello_core"
>> >with this job, so that Katello's test cycle includes the Foreman
>> >tests. The items on our todo list are:
>> >
>> >* Fix the last 8 failures
>> >* Fix the hardcoded SQL migrations to allow testing on other databases
>> >
>> >If anyone has any spare cycles, please take a look at the remaining
>> >failures[3] and see if any of them look familiar. Most appear to be
>> >some kind of change in expected objects, but in particular, one of the
>> >failures is due to how Katello disables the nested behaviour of
>> >Taxonomies, so parent_id cannot be set. This is a behaviour which is
>> >obviously tested in Foreman, and we need to figure out how to handle
>> >it.
>> >
>> >I opened a PR to allow plugins to disable core tests via a custom
>> >test_helper[4] but some people had concerns about the confusion that
>> >might cause. It still feels like the best (where best may not equate
>> >to good :P) solution to me, but I'm open to ideas on how to actually
>> >resolve that failure rather than skip the test. Bear in mind the short
>> >timescales we have before proposing complete re-implementations :P
> Having looked at this with Greg, we either need this test disabling
> functionality, enable org/loc nesting again in Katello or create
> settings that turn org/loc nesting on/off.
>
> Eric
>
Does that matter? It's the Org tests which are failing. The question
is, does it still make sense for Katello to be disabling org nesting?
If not, then we don't have to write a custom test_runner just to allow
a plugin to disable a core test (which has it's own implications for
plugins which might disable a test without replacing it with a new one
see the PR for the discussion).
···
On 7 August 2014 20:56, Bryan Kearney wrote:
> is org _and_ location nesting disabled.. or just org?
Eric and myself have spent some time working on ensuring that the full
Foreman test suite is run with the Katello plugin enabled, so that we
can have greater confidence of the combined result.
There’s now a jenkins job[1] which is currently manual trigger only,
and currently requires Eric’s PR[2] to function. The intention is to
iron out the last few wrinkles and then replace "test_katello_core"
with this job, so that Katello’s test cycle includes the Foreman
tests. The items on our todo list are:
Fix the last 8 failures
Fix the hardcoded SQL migrations to allow testing on other databases
If anyone has any spare cycles, please take a look at the remaining
failures[3] and see if any of them look familiar. Most appear to be
some kind of change in expected objects, but in particular, one of the
failures is due to how Katello disables the nested behaviour of
Taxonomies, so parent_id cannot be set. This is a behaviour which is
obviously tested in Foreman, and we need to figure out how to handle
it.
I opened a PR to allow plugins to disable core tests via a custom
test_helper[4] but some people had concerns about the confusion that
might cause. It still feels like the best (where best may not equate
to good :P) solution to me, but I’m open to ideas on how to actually
resolve that failure rather than skip the test. Bear in mind the short
timescales we have before proposing complete re-implementations
Having looked at this with Greg, we either need this test disabling
functionality, enable org/loc nesting again in Katello or create
settings that turn org/loc nesting on/off.
Eric
is org and location nesting disabled… or just org?
The Foreman core test suite assumes those are on any way - if you
disable them, lots of tests will fail. Individual tests which
temporarily alter the setting should continue to work.
···
On 8 August 2014 12:33, Daniel Lobato wrote:
> Won't this make tests that disable organizations & locations fail? As far as
> I know Katello does need at least one organization.
I have recently started back to working on this in earnest and while I am
close, there are a few observations that require some thought. First, the
current state of this work is captured in a new Foreman PR [1] and updates
to the Katello PR [2]. On the Foreman side, some updates to the test runner
need to be made in order for the test runner to be ran on Jenkins. The only
way I have found to do this is through custom test tasks that override the
Jenkins test runner, see [1] for more details. On the Katello side, the
original pull request [2] has been updated to reduce the number of skipped
tests (thanks in part to the removal of user attributes). If you are
curious to see a mostly green (at least from Foreman core) test run see [3].
Observations
Most of the test that need to be skipped from the following categories:
a) Disablement of organization nesting
b) Katello hooking into host destroy actions
c) Katello needing to hook into Organization CRUD workflow
The test job currently fails on foreman_docker rubocop. Should rubocop
rake tasks be part of the 'unit' test suite?
Katello requires skipping a foreman_docker test because it is a
dependency and enhances the 'jenkins:unit' task
The last observation is what currently has the test job stuck. Because I am
attempting to run 'jenkins:unit', which in Foreman core, runs all of the
Foreman test suites and outputs test reports for CI consumption. Typically
plugins have been built such that they enhance the base test tasks. In the
Rake world, enhancing a task means that you are adding additional tasks to
be run when that task is called. For example, when 'jenkins;unit' is run,
foreman_docker is configured to also have their unit tests and their
rubocop style checking run. This means that if another plugin includes
them, such as Katello, and wants to run the Foreman test suite to verify
those tests still pass with itself present, it will also run the
foreman_docker test suite. I believe there are pros and cons to this
approach.
Pro: ALL tests, core and plugin, that a particular plugin relies on can be
run from a single command and ensure that expectations are met.
Con: A plugin cannot reliably vet itself against the Foreman Core unit
tests only.
Solution Proposal:
Based on the above, my initial proposal would be the generation of two sets
of rake tasks. One set would allow running Foreman core unit tests only and
any enhancing of said tasks would be discouraged (ideally enforceable
through code). The second set would be a supported set of tasks designed to
be enhanced such that all plugins present are contributing their unit tests
to the same bucket to be run with a single command. Another option for the
second set would be a set of tasks that allows declaration of the set to be
run (e.g. rake plugin:tests[docker,katello,discovery].
is org and location nesting disabled… or just org?
Does that matter? It’s the Org tests which are failing. The question
is, does it still make sense for Katello to be disabling org nesting?
If not, then we don’t have to write a custom test_runner just to allow
a plugin to disable a core test (which has it’s own implications for
plugins which might disable a test without replacing it with a new one
