Hi team,
I'm sure most of you remember our good old katello cli smoke test that
were executing commands against katello and recording the results.
I've been missing similar tests since we started to work on hammer and
retired former katello cli. They used to be very good regression check.
Therefore I decided to write something similar some time ago (this time
in ruby).
So far it's been quite useful in catching api and cli divergences so I
decided to share it:
The script is not the cleanest and I hope our ruby gurus won't pelt me
with stones:) I wrote the prototype quickly and didn't bother about it's
structure. The tests are intentionally dependent on each other for the
sake of speed.
What it does:
executes various hammer commands, tests the results and prints stats
at the end
records chunks of foreman's development.log and hammer.log for each
command (it makes digging for the origin of errors easier)
floods your foreman instance with testing data and ruins it with
unexpected failures
AFAIK we don't have anything similar for katello yet. It might be a good
idea to cover at least the basic commands with such tests to be sure
we're not breaking stuff with additional modifications.
katello smoke tests are back, stronger than anytime before: Let's plug this into the CI please.
– Ivan
···
----- Original Message -----
> Hi team,
> I'm sure most of you remember our good old katello cli smoke test that
> were executing commands against katello and recording the results.
>
> I've been missing similar tests since we started to work on hammer and
> retired former katello cli. They used to be very good regression check.
> Therefore I decided to write something similar some time ago (this time
> in ruby).
>
> So far it's been quite useful in catching api and cli divergences so I
> decided to share it:
> https://github.com/tstrachota/hammer-tests
>
> The script is not the cleanest and I hope our ruby gurus won't pelt me
> with stones:) I wrote the prototype quickly and didn't bother about it's
> structure. The tests are intentionally dependent on each other for the
> sake of speed.
>
> What it does:
> - executes various hammer commands, tests the results and prints stats
> at the end
> - records chunks of foreman's development.log and hammer.log for each
> command (it makes digging for the origin of errors easier)
> - floods your foreman instance with testing data and ruins it with
> unexpected failures
>
> AFAIK we don't have anything similar for katello yet. It might be a good
> idea to cover at least the basic commands with such tests to be sure
> we're not breaking stuff with additional modifications.
>
> Cheers
> Tomas
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
There are some tests in foreman-bats1 which AFAIK is run against
nightly. It looks like a very different level of testing, but may be
interesting to be aware of.
···
On Thu, May 15, 2014 at 01:50:44PM +0200, Tomas Strachota wrote:
> Hi team,
> I'm sure most of you remember our good old katello cli smoke test
> that were executing commands against katello and recording the
> results.
>
> I've been missing similar tests since we started to work on hammer
> and retired former katello cli. They used to be very good regression
> check. Therefore I decided to write something similar some time ago
> (this time in ruby).
>
> So far it's been quite useful in catching api and cli divergences so
> I decided to share it:
> https://github.com/tstrachota/hammer-tests
>
> The script is not the cleanest and I hope our ruby gurus won't pelt
> me with stones:) I wrote the prototype quickly and didn't bother
> about it's structure. The tests are intentionally dependent on each
> other for the sake of speed.
>
> What it does:
> - executes various hammer commands, tests the results and prints
> stats at the end
> - records chunks of foreman's development.log and hammer.log for
> each command (it makes digging for the origin of errors easier)
> - floods your foreman instance with testing data and ruins it with
> unexpected failures
>
> AFAIK we don't have anything similar for katello yet. It might be a
> good idea to cover at least the basic commands with such tests to be
> sure we're not breaking stuff with additional modifications.
···
On 15.05.14 13:50, Tomas Strachota wrote:
> Hi team,
> I'm sure most of you remember our good old katello cli smoke test that
> were executing commands against katello and recording the results.
>
> I've been missing similar tests since we started to work on hammer and
> retired former katello cli. They used to be very good regression check.
> Therefore I decided to write something similar some time ago (this time
> in ruby).
>
> So far it's been quite useful in catching api and cli divergences so I
> decided to share it:
> https://github.com/tstrachota/hammer-tests
>
> The script is not the cleanest and I hope our ruby gurus won't pelt me
> with stones:) I wrote the prototype quickly and didn't bother about it's
> structure. The tests are intentionally dependent on each other for the
> sake of speed.
>
> What it does:
> - executes various hammer commands, tests the results and prints stats
> at the end
> - records chunks of foreman's development.log and hammer.log for each
> command (it makes digging for the origin of errors easier)
> - floods your foreman instance with testing data and ruins it with
> unexpected failures
>
> AFAIK we don't have anything similar for katello yet. It might be a good
> idea to cover at least the basic commands with such tests to be sure
> we're not breaking stuff with additional modifications.
>
> Cheers
> Tomas
>
I have been using hammer-cli-csv[1] for testing the api and some cli. With the constant flux in both, however, it is lagging by a few weeks (ie. I'm not sure what is out of date ;). I use the setup_test.rb to populate my "Mega Corporation" with data so it hits much of the index/show/create api.
Here is an example test:
it "hammer -v host info --id $id" do
stdout,stderr = capture {
hammer.run(%W{-v host info --id #{host_id}})
}
stderr.must_equal ''
attributes = {}
stdout.split("\n").each do |line|
name,value = line.split(":")
attributes[name.strip] = (value.nil? ? '' : value.strip)
end
attributes['Id'].must_equal host_id
attributes['Name'].must_equal 'dhcp129-000.megacorp.com'
end
···
----- Original Message -----
> On Thu, May 15, 2014 at 01:50:44PM +0200, Tomas Strachota wrote:
> > Hi team,
> > I'm sure most of you remember our good old katello cli smoke test
> > that were executing commands against katello and recording the
> > results.
> >
> > I've been missing similar tests since we started to work on hammer
> > and retired former katello cli. They used to be very good regression
> > check. Therefore I decided to write something similar some time ago
> > (this time in ruby).
> >
> > So far it's been quite useful in catching api and cli divergences so
> > I decided to share it:
> > https://github.com/tstrachota/hammer-tests
> >
> > The script is not the cleanest and I hope our ruby gurus won't pelt
> > me with stones:) I wrote the prototype quickly and didn't bother
> > about it's structure. The tests are intentionally dependent on each
> > other for the sake of speed.
> >
> > What it does:
> > - executes various hammer commands, tests the results and prints
> > stats at the end
> > - records chunks of foreman's development.log and hammer.log for
> > each command (it makes digging for the origin of errors easier)
> > - floods your foreman instance with testing data and ruins it with
> > unexpected failures
> >
> > AFAIK we don't have anything similar for katello yet. It might be a
> > good idea to cover at least the basic commands with such tests to be
> > sure we're not breaking stuff with additional modifications.
>
> There are some tests in foreman-bats[1] which AFAIK is run against
> nightly. It looks like a very different level of testing, but may be
> interesting to be aware of.
>
> [1]:
> https://github.com/theforeman/foreman-bats/blob/master/fb-install-foreman.bats#L148-L162
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
Yep, I know about bats. They're great but they test the installation
process and verify the setup afterwards. Purpose of hammer-tests is a
bit different - to test functionality of both hammer and the api as a whole.
T.
···
On 05/15/2014 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
> On Thu, May 15, 2014 at 01:50:44PM +0200, Tomas Strachota wrote:
>> Hi team,
>> I'm sure most of you remember our good old katello cli smoke test
>> that were executing commands against katello and recording the
>> results.
>>
>> I've been missing similar tests since we started to work on hammer
>> and retired former katello cli. They used to be very good regression
>> check. Therefore I decided to write something similar some time ago
>> (this time in ruby).
>>
>> So far it's been quite useful in catching api and cli divergences so
>> I decided to share it:
>> https://github.com/tstrachota/hammer-tests
>>
>> The script is not the cleanest and I hope our ruby gurus won't pelt
>> me with stones:) I wrote the prototype quickly and didn't bother
>> about it's structure. The tests are intentionally dependent on each
>> other for the sake of speed.
>>
>> What it does:
>> - executes various hammer commands, tests the results and prints
>> stats at the end
>> - records chunks of foreman's development.log and hammer.log for
>> each command (it makes digging for the origin of errors easier)
>> - floods your foreman instance with testing data and ruins it with
>> unexpected failures
>>
>> AFAIK we don't have anything similar for katello yet. It might be a
>> good idea to cover at least the basic commands with such tests to be
>> sure we're not breaking stuff with additional modifications.
>
> There are some tests in foreman-bats[1] which AFAIK is run against
> nightly. It looks like a very different level of testing, but may be
> interesting to be aware of.
>
> [1]: https://github.com/theforeman/foreman-bats/blob/master/fb-install-foreman.bats#L148-L162
>
I realize that foreman-bats is designed to test a different aspect
(installation and basic server health post-install) but the over-arching
goal is the same - a set of integration tests. As I understand it, this is
the same rough goal of Robottelo. We've been taking great strides in a lot
of areas to converge on the same designs, infrastructure, code and
mechanisms. Let us not forget this goal, and take the time to figure out
how we can ensure we are not repeating ourselves and are converging on a
consistent way of attacking an area of the project.
Eric
···
On Thu, May 15, 2014 at 1:16 PM, Tomas Strachota wrote:
On 05/15/2014 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
On Thu, May 15, 2014 at 01:50:44PM +0200, Tomas Strachota wrote:
Hi team,
I’m sure most of you remember our good old katello cli smoke test
that were executing commands against katello and recording the
results.
I’ve been missing similar tests since we started to work on hammer
and retired former katello cli. They used to be very good regression
check. Therefore I decided to write something similar some time ago
(this time in ruby).
The script is not the cleanest and I hope our ruby gurus won’t pelt
me with stones:) I wrote the prototype quickly and didn’t bother
about it’s structure. The tests are intentionally dependent on each
other for the sake of speed.
What it does:
executes various hammer commands, tests the results and prints
stats at the end
records chunks of foreman’s development.log and hammer.log for
each command (it makes digging for the origin of errors easier)
floods your foreman instance with testing data and ruins it with
unexpected failures
AFAIK we don’t have anything similar for katello yet. It might be a
good idea to cover at least the basic commands with such tests to be
sure we’re not breaking stuff with additional modifications.
There are some tests in foreman-bats1 which AFAIK is run against
nightly. It looks like a very different level of testing, but may be
interesting to be aware of.
install-foreman.bats#L148-L162
Yep, I know about bats. They’re great but they test the installation
process and verify the setup afterwards. Purpose of hammer-tests is a bit
different - to test functionality of both hammer and the api as a whole.
I knew about robotello in the time I was writing the hammer-test and I
was actually using it before. I don't think hammer-tests are replacement
for robotello at all. Robotello tests much more corner cases and is more
robust tool that suits QEs needs.
My motivation was to create a simple script that suits my personal devel
needs, quickly fires some commands, tests if I didn't break anything
with my patches and tells me what went wrong and where. It worked quite
well for me so I decided to share it.
Please, don't get me wrong, I'm not forcing anybody to start using it.
Peace
T.
···
On 05/16/2014 04:15 AM, Eric D Helms wrote:
I realize that foreman-bats is designed to test a different aspect
(installation and basic server health post-install) but the over-arching
goal is the same - a set of integration tests. As I understand it, this
is the same rough goal of Robottelo. We’ve been taking great strides in
a lot of areas to converge on the same designs, infrastructure, code and
mechanisms. Let us not forget this goal, and take the time to figure out
how we can ensure we are not repeating ourselves and are converging on a
consistent way of attacking an area of the project.
On 05/15/2014 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
On Thu, May 15, 2014 at 01:50:44PM +0200, Tomas Strachota wrote:
Hi team,
I'm sure most of you remember our good old katello cli smoke
test
that were executing commands against katello and recording the
results.
I've been missing similar tests since we started to work on
hammer
and retired former katello cli. They used to be very good
regression
check. Therefore I decided to write something similar some
time ago
(this time in ruby).
So far it's been quite useful in catching api and cli
divergences so
I decided to share it:
https://github.com/tstrachota/__hammer-tests
<https://github.com/tstrachota/hammer-tests>
The script is not the cleanest and I hope our ruby gurus
won't pelt
me with stones:) I wrote the prototype quickly and didn't bother
about it's structure. The tests are intentionally dependent
on each
other for the sake of speed.
What it does:
- executes various hammer commands, tests the results and prints
stats at the end
- records chunks of foreman's development.log and hammer.log for
each command (it makes digging for the origin of errors easier)
- floods your foreman instance with testing data and ruins
it with
unexpected failures
AFAIK we don't have anything similar for katello yet. It
might be a
good idea to cover at least the basic commands with such
tests to be
sure we're not breaking stuff with additional modifications.
There are some tests in foreman-bats[1] which AFAIK is run against
nightly. It looks like a very different level of testing, but may be
interesting to be aware of.
[1]:
https://github.com/theforeman/__foreman-bats/blob/master/fb-__install-foreman.bats#L148-L162
<https://github.com/theforeman/foreman-bats/blob/master/fb-install-foreman.bats#L148-L162>
Yep, I know about bats. They're great but they test the installation
process and verify the setup afterwards. Purpose of hammer-tests is
a bit different - to test functionality of both hammer and the api
as a whole.
T.
--
You received this message because you are subscribed to the Google
Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to foreman-dev+unsubscribe@__googlegroups.com
<mailto:foreman-dev%2Bunsubscribe@googlegroups.com>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
I was unaware of robottelo at the time we wrote foreman-bats. My
personal use case for foreman-bats is testing if the installer modules
work well and result in a working setup. This means I want to be able to
spin up a VM, run the installer (possibly with a different set of
modules to test development branches) and verify I have a working
system.
An alternative would be to use beaker1 to do acceptance tests in
foreman-installer and/or every single module. When we wrote
foreman-bats, it wasn't that mature (lacking documentation for example)
but I am considering writing some tests as a proof of concept.
That said, I am highly in favor of concentrating efforts to do testing.
In the short term could we simply add a step to run robottelo as part of
foreman-bats and have the best of both worlds?
···
On Thu, May 15, 2014 at 10:15:24PM -0400, Eric D Helms wrote:
> How does this differ from https://github.com/omaciel/robottelo ? Are we
> once again repeating ourselves and wasting efforts that could be converged
> on one or the other?
>
> I realize that foreman-bats is designed to test a different aspect
> (installation and basic server health post-install) but the over-arching
> goal is the same - a set of integration tests. As I understand it, this is
> the same rough goal of Robottelo. We've been taking great strides in a lot
> of areas to converge on the same designs, infrastructure, code and
> mechanisms. Let us not forget this goal, and take the time to figure out
> how we can ensure we are not repeating ourselves and are converging on a
> consistent way of attacking an area of the project.
>
> Eric
>
>
> On Thu, May 15, 2014 at 1:16 PM, Tomas Strachota wrote:
>
> > On 05/15/2014 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
> >
> >> On Thu, May 15, 2014 at 01:50:44PM +0200, Tomas Strachota wrote:
> >>
> >>> Hi team,
> >>> I'm sure most of you remember our good old katello cli smoke test
> >>> that were executing commands against katello and recording the
> >>> results.
> >>>
> >>> I've been missing similar tests since we started to work on hammer
> >>> and retired former katello cli. They used to be very good regression
> >>> check. Therefore I decided to write something similar some time ago
> >>> (this time in ruby).
> >>>
> >>> So far it's been quite useful in catching api and cli divergences so
> >>> I decided to share it:
> >>> https://github.com/tstrachota/hammer-tests
> >>>
> >>> The script is not the cleanest and I hope our ruby gurus won't pelt
> >>> me with stones:) I wrote the prototype quickly and didn't bother
> >>> about it's structure. The tests are intentionally dependent on each
> >>> other for the sake of speed.
> >>>
> >>> What it does:
> >>> - executes various hammer commands, tests the results and prints
> >>> stats at the end
> >>> - records chunks of foreman's development.log and hammer.log for
> >>> each command (it makes digging for the origin of errors easier)
> >>> - floods your foreman instance with testing data and ruins it with
> >>> unexpected failures
> >>>
> >>> AFAIK we don't have anything similar for katello yet. It might be a
> >>> good idea to cover at least the basic commands with such tests to be
> >>> sure we're not breaking stuff with additional modifications.
> >>>
> >>
> >> There are some tests in foreman-bats[1] which AFAIK is run against
> >> nightly. It looks like a very different level of testing, but may be
> >> interesting to be aware of.
> >>
> >> [1]: https://github.com/theforeman/foreman-bats/blob/master/fb-
> >> install-foreman.bats#L148-L162
> >>
> >>
> > Yep, I know about bats. They're great but they test the installation
> > process and verify the setup afterwards. Purpose of hammer-tests is a bit
> > different - to test functionality of both hammer and the api as a whole.
> >
> > T.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "foreman-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to foreman-dev+unsubscribe@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
