Today, a number of plugins run the full Foreman core test suite as part of
their test runs by enhancing the jenkins:unit Rake task. Katello, being
more complex than most, requires a few different plugins by default and
thus to run it's tests plus Foreman's core and the plugins has required
some work to allow skipping some tests when the combination is available.
This email is geared around discussing our overall strategy in this area.
I'll start with what is essentially the status quo.
Individual Plugin Tests
All plugins should enhance the jenkins:unit test so that when running their
test suite on our CI system, their tests + Foreman core + any dependent
plugins' tests are run.
An example of this can be taken from foreman-docker:
if Rake::Task.defined?('jenkins:unit')
Rake::Task['jenkins:unit'].enhance do
Rake::Task['test:docker'].invoke
Rake::Task['docker:rubocop'].invoke
end
end
Questions:
Do we simply document this as a suggested strategy for plugins to take?
Do we create a rake task in Foreman itself to enforce this method or
update the jenkins rake tasks such that they do?
Multi-Plugin Test Runs
The multi-plugin test run would be a job that allows running the test suite
of multiple plugins enabled at the same time. This could manifest as a
single job that has options to enable or disable a set of known plugins
before running the test suite. The default could be that our 'supported'
plugins are always enabled in the base configuration for the job. That
leads to the question of when and where do we run this job so make use of
it's output within our CI workflows/pipelines.
···
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
> Today, a number of plugins run the full Foreman core test suite as part
> of their test runs by enhancing the jenkins:unit Rake task. Katello,
> being more complex than most, requires a few different plugins by
> default and thus to run it's tests plus Foreman's core and the plugins
> has required some work to allow skipping some tests when the combination
> is available. This email is geared around discussing our overall
> strategy in this area. I'll start with what is essentially the status quo.
>
> Individual Plugin Tests
>
> All plugins should enhance the jenkins:unit test so that when running
> their test suite on our CI system, their tests + Foreman core + any
> dependent plugins' tests are run.
>
[snip]
> 1) Do we simply document this as a suggested strategy for plugins to take?
It kind of is, by virtue of it being in the example plugin.
> 2) Do we create a rake task in Foreman itself to enforce this method or
> update the jenkins rake tasks such that they do?
Not sure that I understand precisely what you mean?
> Multi-Plugin Test Runs
> *
> *
> The multi-plugin test run would be a job that allows running the test
> suite of multiple plugins enabled at the same time. This could manifest
> as a single job that has options to enable or disable a set of known
> plugins before running the test suite. The default could be that our
> 'supported' plugins are always enabled in the base configuration for the
> job. That leads to the question of when and where do we run this job so
> make use of it's output within our CI workflows/pipelines.
I guess you could just have a scheduled job, once a week or something
that runs a fixed set of everything that's tested in Jenkins already
together in one job.
> Today, a number of plugins run the full Foreman core test suite as part of
> their test runs by enhancing the jenkins:unit Rake task. Katello, being
> more complex than most, requires a few different plugins by default and
> thus to run it's tests plus Foreman's core and the plugins has required
> some work to allow skipping some tests when the combination is available.
> This email is geared around discussing our overall strategy in this area.
> I'll start with what is essentially the status quo.
>
> Individual Plugin Tests
>
> All plugins should enhance the jenkins:unit test so that when running their
> test suite on our CI system, their tests + Foreman core + any dependent
> plugins' tests are run.
>
> An example of this can be taken from foreman-docker:
>
> if Rake::Task.defined?('jenkins:unit')
> Rake::Task['jenkins:unit'].enhance do
> Rake::Task['test:docker'].invoke
> Rake::Task['docker:rubocop'].invoke
> end
> end
>
> Questions:
>
> 1) Do we simply document this as a suggested strategy for plugins to take?
> 2) Do we create a rake task in Foreman itself to enforce this method or
> update the jenkins rake tasks such that they do?
>
> Multi-Plugin Test Runs
>
> The multi-plugin test run would be a job that allows running the test suite
> of multiple plugins enabled at the same time. This could manifest as a
> single job that has options to enable or disable a set of known plugins
> before running the test suite. The default could be that our 'supported'
> plugins are always enabled in the base configuration for the job. That
> leads to the question of when and where do we run this job so make use of
> it's output within our CI workflows/pipelines.
I think I understand the proposal (make plugins extend a new task,
jenkins:plugins and run that), where I'm lost is trying to understand
what would be the benefit? It's slighly more complicated as we'd have to
run both tasks in Jenkins.
···
On 07/09, Eric D Helms wrote:
–
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
> > Today, a number of plugins run the full Foreman core test suite as part
> > of their test runs by enhancing the jenkins:unit Rake task. Katello,
> > being more complex than most, requires a few different plugins by
> > default and thus to run it's tests plus Foreman's core and the plugins
> > has required some work to allow skipping some tests when the combination
> > is available. This email is geared around discussing our overall
> > strategy in this area. I'll start with what is essentially the status
> quo.
> >
> > Individual Plugin Tests
> >
> > All plugins should enhance the jenkins:unit test so that when running
> > their test suite on our CI system, their tests + Foreman core + any
> > dependent plugins' tests are run.
> >
> [snip]
> > 1) Do we simply document this as a suggested strategy for plugins to
> take?
>
> It kind of is, by virtue of it being in the example plugin.
>
> > 2) Do we create a rake task in Foreman itself to enforce this method or
> > update the jenkins rake tasks such that they do?
>
> Not sure that I understand precisely what you mean?
>
I was thinking something where when the jenkins:unit task is run, there
is some detection of enabled plugins and it adds their tests or test take
into the mix. This would rely on either a commonly named test task for
plugins (e.g. rake test:plugin:<plugin_name>) or on an assumption that
plugins will all have a test/ directory.
···
On Thu, Jul 9, 2015 at 10:34 AM, Dominic Cleal wrote:
> On 09/07/15 15:23, Eric D Helms wrote:
Multi-Plugin Test Runs
*
*
The multi-plugin test run would be a job that allows running the test
suite of multiple plugins enabled at the same time. This could manifest
as a single job that has options to enable or disable a set of known
plugins before running the test suite. The default could be that our
’supported’ plugins are always enabled in the base configuration for the
job. That leads to the question of when and where do we run this job so
make use of it’s output within our CI workflows/pipelines.
I guess you could just have a scheduled job, once a week or something
that runs a fixed set of everything that’s tested in Jenkins already
together in one job.
Yeah, we could do that - or have plugins give their rake task name(s)
via the plugin registration. Either way would make it easy for core to
add all plugins to the main 'test' and 'jenkins:unit' tasks, which is
probably a better idea than every plugin trying to extend the task(s)
they know about.
We could also go in the other direction and never automatically extend
or run tests from plugins, and only do it explicitly. From the PoV of
our Jenkins tests, this would be trivial to do still (we'd just put the
rake task names in the job descs) and would also make it easier to do
say "run foreman_docker tests without foreman-tasks tests, but with
Foreman core tests".
···
On 09/07/15 15:52, Eric D Helms wrote:
>
>
> On Thu, Jul 9, 2015 at 10:34 AM, Dominic Cleal > wrote:
>
> On 09/07/15 15:23, Eric D Helms wrote:
> > Today, a number of plugins run the full Foreman core test suite as part
> > of their test runs by enhancing the jenkins:unit Rake task. Katello,
> > being more complex than most, requires a few different plugins by
> > default and thus to run it's tests plus Foreman's core and the plugins
> > has required some work to allow skipping some tests when the combination
> > is available. This email is geared around discussing our overall
> > strategy in this area. I'll start with what is essentially the status quo.
> >
> > *Individual Plugin Tests*
> >
> > All plugins should enhance the jenkins:unit test so that when running
> > their test suite on our CI system, their tests + Foreman core + any
> > dependent plugins' tests are run.
> >
> [snip]
> > 1) Do we simply document this as a suggested strategy for plugins to take?
>
> It kind of is, by virtue of it being in the example plugin.
>
> > 2) Do we create a rake task in Foreman itself to enforce this method or
> > update the jenkins rake tasks such that they do?
>
> Not sure that I understand precisely what you mean?
>
>
> I was thinking something where when the `jenkins:unit` task is run,
> there is some detection of enabled plugins and it adds their tests or
> test take into the mix. This would rely on either a commonly named test
> task for plugins (e.g. rake test:plugin:) or on an
> assumption that plugins will all have a test/ directory.
