From a release engineer perspective I’d prefer to have something. That also means I wouldn’t mind a temporary solution, even if we throw it away after a few months. My requirements:
- Black box. It should have nothing on the actual system under test. That means providing it with an url, username and password should be sufficient.
- An easy traffic light functionality: red if it’s broken, green if it’s working
Nice to have:
- An easy to consume test output. A common format is xunit but Jenkins can also easily test TAP
- Flexible. It should work on a vanilla Foreman, but also handle plugins.
- Quick.
- Have a multi-version strategy. I’d be ok with just using this for nightly, but it’d be great if we could also use it on releases.