I'm starting this thread to talk about getting help from the community and
devs to help improving the test coverage quickly.
We develop Robottelo and it offers tools to help testing Foreman features.
It still dependent of the infrastructure for the QE team, but it should be
decoupled in the future to allow any foreman dev to create and run the
tests.
Some main issues we may have is being able to tests compute resources and
other features that depends on some infrastructure.
I don't have a plan yet, but I think we can build a better plan together.
> I'm starting this thread to talk about getting help from the community and
> devs to help improving the test coverage quickly.
>
> We develop Robottelo and it offers tools to help testing Foreman features.
> It still dependent of the infrastructure for the QE team, but it should be
> decoupled in the future to allow any foreman dev to create and run the
> tests.
>
> Some main issues we may have is being able to tests compute resources and
> other features that depends on some infrastructure.
>
> I don't have a plan yet, but I think we can build a better plan together.
>
I would be interested to compare robottelo vs what we currently have in ci.theforeman.org.
What would it take for us to run robetllo in ci.theforeman.org ?
We are currently consuming resources via loans, rackspace, and hopefully in
the future resources from the centos project as well.
if we could unify it under one jenkins infrastructure, i think it could
unify and focus our efforts?
Ohad
···
On Wed, Aug 5, 2015 at 4:32 PM, Elyézer Rezende wrote:
> All,
>
> I'm starting this thread to talk about getting help from the community and
> devs to help improving the test coverage quickly.
>
> We develop Robottelo and it offers tools to help testing Foreman features.
> It still dependent of the infrastructure for the QE team, but it should be
> decoupled in the future to allow any foreman dev to create and run the
> tests.
>
> Some main issues we may have is being able to tests compute resources and
> other features that depends on some infrastructure.
Ideally we could get 'small' containerized hypervisors for Openstack,
Ovirt, Libvirt, Xen, Vmware maybe. Do these tests test anything on
GCE/AWS/Rackspace/Digitalocean compute resources? Those would be the
hardest ones to automate (for free I mean)
···
On 08/05, Elyézer Rezende wrote:
I don’t have a plan yet, but I think we can build a better plan together.
More robust automated system and integration testing that covers
API/CLI/UI (seems like Robottelo brings a lot of this)
Testing resources (i.e. hardware)
#1 would allow us to catch issues sooner and reduce the overall costs of
defects. #2 would allow us to both run more verbose test sets and release
in a more continuous fashion. At present we run the release pipelines once
a day due to resource concerns which means the delta of changes is greater
and if the pipeline breaks there is a longer window before it potentially
gets re-tested and fixed which can then lead to pile-up as we have seen
over the past week or so.
Eric
···
From my experience, our two biggest concerns are:
On Wed, Aug 5, 2015 at 4:32 PM, Elyézer Rezende elyezermr@gmail.com > wrote:
All,
thanks for starting the thread!
I’m starting this thread to talk about getting help from the community
and devs to help improving the test coverage quickly.
We develop Robottelo and it offers tools to help testing Foreman
features. It still dependent of the infrastructure for the QE team, but it
should be decoupled in the future to allow any foreman dev to create and
run the tests.
Some main issues we may have is being able to tests compute resources and
other features that depends on some infrastructure.
I don’t have a plan yet, but I think we can build a better plan together.
I would be interested to compare robottelo vs what we currently have in ci.theforeman.org.
What would it take for us to run robetllo in ci.theforeman.org ?
We are currently consuming resources via loans, rackspace, and hopefully
in the future resources from the centos project as well.
if we could unify it under one jenkins infrastructure, i think it could
unify and focus our efforts?
···
On 08/06/2015 04:41 PM, Eric D Helms wrote:
> From my experience, our two biggest concerns are:
>
> 1) More robust automated system and integration testing that covers
> API/CLI/UI (seems like Robottelo brings a lot of this)
> 2) Testing resources (i.e. hardware)
>
> #1 would allow us to catch issues sooner and reduce the overall costs of
> defects. #2 would allow us to both run more verbose test sets and
> release in a more continuous fashion. At present we run the release
> pipelines once a day due to resource concerns which means the delta of
> changes is greater and if the pipeline breaks there is a longer window
> before it potentially gets re-tested and fixed which can then lead to
> pile-up as we have seen over the past week or so.
>
···
On 12/08/15 18:38, Bryan Kearney wrote:
> On 08/06/2015 04:41 PM, Eric D Helms wrote:
>> From my experience, our two biggest concerns are:
>>
>> 1) More robust automated system and integration testing that covers
>> API/CLI/UI (seems like Robottelo brings a lot of this)
>> 2) Testing resources (i.e. hardware)
>>
>> #1 would allow us to catch issues sooner and reduce the overall costs of
>> defects. #2 would allow us to both run more verbose test sets and
>> release in a more continuous fashion. At present we run the release
>> pipelines once a day due to resource concerns which means the delta of
>> changes is greater and if the pipeline breaks there is a longer window
>> before it potentially gets re-tested and fixed which can then lead to
>> pile-up as we have seen over the past week or so.
>>
>
> What type of hardware do you need?
···
On 08/13/2015 06:02 AM, Dominic Cleal wrote:
> Regular Jenkins slaves are always very welcome, which would allow us to
> use more of our Rackspace Cloud allowance for one-off tests.
>
> http://projects.theforeman.org/projects/foreman/wiki/Jenkins#section-25
> lists the requirements.
Especially if we are adding more system testing (i.e. Robotello), we, as
Dominic mentions, either need more Jenkins slaves so we can use Rackspace
for those system tests or to add hardware that we can similarly provision
to (e.g. a box(es) with libvirt we can run vagrant bats tests and future
system tests on). WIth nightlies being run once a day, and trying to limit
how often we run them out of band it can drag out the fix/release cycle
when things in the pipeline do break for one reason or another.
···
On Thu, Aug 13, 2015 at 8:13 AM, Bryan Kearney wrote:
On 08/13/2015 06:02 AM, Dominic Cleal wrote:
Regular Jenkins slaves are always very welcome, which would allow us to
use more of our Rackspace Cloud allowance for one-off tests.
The jenkins jobs that is used for running robotello automation is available
on [1].
I think we can start by defining how a production infrastructure would be
if want to use all compute resources and provisioning stuff. Then we can
check if we can get this entire infrastructure ready to run the tests or
work with will be available. This infrastructure scheme will also help
defining Robottelo path in order to provide tools to help writing tests as
it currently test libvirt and docker compute resources.
Some parts of robottelo relies on the current infrastructure used, it will
require some changes and also provide more configuration options.
I will take a look on the current ci.theforeman.org structure and jobs to
check how it works currently.
···
On Thu, Aug 13, 2015 at 7:02 AM Dominic Cleal wrote:
On 12/08/15 18:38, Bryan Kearney wrote:
On 08/06/2015 04:41 PM, Eric D Helms wrote:
From my experience, our two biggest concerns are:
More robust automated system and integration testing that covers
API/CLI/UI (seems like Robottelo brings a lot of this)
Testing resources (i.e. hardware)
#1 would allow us to catch issues sooner and reduce the overall costs of
defects. #2 would allow us to both run more verbose test sets and
release in a more continuous fashion. At present we run the release
pipelines once a day due to resource concerns which means the delta of
changes is greater and if the pipeline breaks there is a longer window
before it potentially gets re-tested and fixed which can then lead to
pile-up as we have seen over the past week or so.
What type of hardware do you need?
Regular Jenkins slaves are always very welcome, which would allow us to
use more of our Rackspace Cloud allowance for one-off tests.
The way I've been trying to use it is as a way to spin up a hypervisor
which we can connect to with vagrant-libvirt, so we retain some
abstraction and can reuse existing tools and tests. I hope to have
something usable in the next few weeks, and can perhaps then easily
point central jobs like systest_foreman at it. I'll post some more
details soon.
There may be some sort of OpenStack instance available on the same infra
in future, which would be be even better, giving a more similar
experience to vagrant-rackspace.
···
On 13/08/15 13:23, Eric D Helms wrote:
> Especially if we are adding more system testing (i.e. Robotello), we, as
> Dominic mentions, either need more Jenkins slaves so we can use
> Rackspace for those system tests or to add hardware that we can
> similarly provision to (e.g. a box(es) with libvirt we can run vagrant
> bats tests and future system tests on).
