I've recently added a few more systest_foreman_* jobs to Jenkins which
test things other than core package smoke tests.
The recent additions are the MySQL installation test, the move of the
Puppet integration tests out of smoke tests, and the umask 077 test. An
upgrade test is probably the next important one to write in foreman-bats
(help welcome!).
They remain separate to smoke tests so as not to hold up publishing of
nightly packages with non-critical bugs and to reduce the infrastructure
load (jobs and budget). Their failures are important though, so should
be resolved in time for releases. They usually run once or twice a week
around 7-9am UTC, and print to #theforeman-dev.
I also changed the tests to use the published rather than unpublished
packages today, so you should see fewer transient failures.
There's more info about the setup on the wiki
(Jenkins - Foreman)
and the jobs are all on this view: http://ci.theforeman.org/view/Status/.
I'm also looking at how to change systest_foreman to run our jobs on the
CentOS CI infrastructure which will cut down our use of Rackspace for
these short jobs, perhaps allowing us to run more permanent slaves for
regular unit tests.
Cheers,