Some new testing of RPMs published to yum.theforeman.org has been added
to Jenkins. It uses the foreman-bats project started by lzap to run a
basic smoke test of our packages, the installer and hammer:
Jenkins is configured to run this as part of the RPM "publish" process
(the packaging_publish_rpm job), so a repo is created on koji (e.g. the
nightly or a release repo), this job is called and runs repoclosure,
then foreman-bats and then publishes it to yum.tf.org if successful.
The Jenkins job is using Vagrant and the vagrant-rackspace plugin to run
foreman-bats on the Rackspace public cloud. Many thanks to Rackspace,
who have donated this account to the project.
Some more info about the setup and links are here:
This isn't a replacement for tests in our projects, but if there are
additional smoke tests that we should include, please send PRs to
foreman-bats. Hopefully it'll increase the quality of our nightly
builds, and also save us a lot of manual smoke testing when doing releases.
The same is nearly ready for Debian as a nightly testing job (see
systest_foreman_debian and #3934).
I'll show this in tomorrow's sprint demo too.