Is there a resource available to the Foreman project that would allow
theforeman Puppet modules to have beaker-rspec acceptance tests run for
regression testing? I know that Jenkins exists for regression testing
Foreman pull requests, but can that same instance be configured to spawn
Vagrant VMs for testing modules using beaker-rspec? I'd be happy to
initiate the pull requests on the various modules (time permitting) to get
the code committed, but it would only be really useful if all commits
initiate the acceptance tests on some CI system (Jenkins, etc).
I have tested spawning tests that span multiple VMs, and that utilize
PuppetDB with a single Puppetmaster, so theoretically testing the setup of
Foreman, proxies, etc should be possible.
I'm in favour of using beaker-rspec on these modules, it works pretty
well these days. We can add them to our Jenkins instance for
post-commit tests pretty easily, though PR testing would be quite a bit
harder (especially as we already use Travis).
My only minor concern about beaker-rspec is actually the nodesets when
using Vagrant, as the implicit assumption in most modules seems to be
that you're using "standard" VirtualBox boxes. As long as we can add
nodesets for both vagrant-libvirt and vagrant-rackspace, we should be
able to use it both from Jenkins (uses Rackspace Cloud) and for those of
us who are on libvirt. (Box names are in the foreman-bats project that
Ewoud mentioned.)
···
On 03/06/14 17:48, Trey Dockendorf wrote:
> Is there a resource available to the Foreman project that would allow
> theforeman Puppet modules to have beaker-rspec acceptance tests run for
> regression testing? I know that Jenkins exists for regression testing
> Foreman pull requests, but can that same instance be configured to spawn
> Vagrant VMs for testing modules using beaker-rspec? I'd be happy to
> initiate the pull requests on the various modules (time permitting) to
> get the code committed, but it would only be really useful if all
> commits initiate the acceptance tests on some CI system (Jenkins, etc).
Currently there is nothing based on beaker-rspec, but we do have
foreman-bats1 which is aimed at doing full integration testing. Sadly
there's only a simple all-in-one installation there that gets tested,
but multi VM testing is something that ought to be there.
Personally I'm fine with beaker-rspec, but some are more in favor of
bats. I've also thought about adding beaker-rspec tests per module so we
get somewhat more fine grained testing than a full installation.
In short, I highly welcome more automated testing and would at the very
least like to help you review and answer any of the questions you might
have.
···
On Tue, Jun 03, 2014 at 09:48:56AM -0700, Trey Dockendorf wrote:
> Is there a resource available to the Foreman project that would allow
> theforeman Puppet modules to have beaker-rspec acceptance tests run for
> regression testing? I know that Jenkins exists for regression testing
> Foreman pull requests, but can that same instance be configured to spawn
> Vagrant VMs for testing modules using beaker-rspec? I'd be happy to
> initiate the pull requests on the various modules (time permitting) to get
> the code committed, but it would only be really useful if all commits
> initiate the acceptance tests on some CI system (Jenkins, etc).
>
> I have tested spawning tests that span multiple VMs, and that utilize
> PuppetDB with a single Puppetmaster, so theoretically testing the setup of
> Foreman, proxies, etc should be possible.
> > Is there a resource available to the Foreman project that would allow
> > theforeman Puppet modules to have beaker-rspec acceptance tests run for
> > regression testing? I know that Jenkins exists for regression testing
> > Foreman pull requests, but can that same instance be configured to spawn
> > Vagrant VMs for testing modules using beaker-rspec? I'd be happy to
> > initiate the pull requests on the various modules (time permitting) to
> > get the code committed, but it would only be really useful if all
> > commits initiate the acceptance tests on some CI system (Jenkins, etc).
>
> I'm in favour of using beaker-rspec on these modules, it works pretty
> well these days. We can add them to our Jenkins instance for
> post-commit tests pretty easily, though PR testing would be quite a bit
> harder (especially as we already use Travis).
I'm on the same page here.
> My only minor concern about beaker-rspec is actually the nodesets when
> using Vagrant, as the implicit assumption in most modules seems to be
> that you're using "standard" VirtualBox boxes. As long as we can add
> nodesets for both vagrant-libvirt and vagrant-rackspace, we should be
> able to use it both from Jenkins (uses Rackspace Cloud) and for those of
> us who are on libvirt. (Box names are in the foreman-bats project that
> Ewoud mentioned.)
As we discussed on IRC: beaker-rspec also supports docker as a backend.
Few minor gotchas, but easy to get up and running. Since it's doesn't
require virtual machines, maybe it can run on Travis as well.
···
On Wed, Jun 04, 2014 at 08:48:17AM +0100, Dominic Cleal wrote:
> On 03/06/14 17:48, Trey Dockendorf wrote:
(tito's sort of using it for their Docker-based tests.)
···
On 04/06/14 11:45, Ewoud Kohl van Wijngaarden wrote:
>> > My only minor concern about beaker-rspec is actually the nodesets when
>> > using Vagrant, as the implicit assumption in most modules seems to be
>> > that you're using "standard" VirtualBox boxes. As long as we can add
>> > nodesets for both vagrant-libvirt and vagrant-rackspace, we should be
>> > able to use it both from Jenkins (uses Rackspace Cloud) and for those of
>> > us who are on libvirt. (Box names are in the foreman-bats project that
>> > Ewoud mentioned.)
> As we discussed on IRC: beaker-rspec also supports docker as a backend.
> Few minor gotchas, but easy to get up and running. Since it's doesn't
> require virtual machines, maybe it can run on Travis as well.
···
On Wed, Jun 04, 2014 at 11:48:32AM +0100, Dominic Cleal wrote:
> On 04/06/14 11:45, Ewoud Kohl van Wijngaarden wrote:
> >> > My only minor concern about beaker-rspec is actually the nodesets when
> >> > using Vagrant, as the implicit assumption in most modules seems to be
> >> > that you're using "standard" VirtualBox boxes. As long as we can add
> >> > nodesets for both vagrant-libvirt and vagrant-rackspace, we should be
> >> > able to use it both from Jenkins (uses Rackspace Cloud) and for those of
> >> > us who are on libvirt. (Box names are in the foreman-bats project that
> >> > Ewoud mentioned.)
> > As we discussed on IRC: beaker-rspec also supports docker as a backend.
> > Few minor gotchas, but easy to get up and running. Since it's doesn't
> > require virtual machines, maybe it can run on Travis as well.
>
> As far as I know, it's not possible to run Docker on Travis for some
> reason, but there is a similar service called Wercker which can.
>
> https://app.wercker.com/#explore
>
> (tito's sort of using it for their Docker-based tests.)
