New repository for smoke tests

[John_Mitsch]

@ekohl I think its a great idea to have release-based smoke testing automation, especially fitting our immediate need to test for blank pages.

I wonder if we need another respository for these tests? Could this all live in forklift? We have so many repositories as it is, IMHO it can often be confusing and hard to find bits of code when things are so spread out. I think we should ask if there is any existing place these smoker tests can live before creating a new repository.

@TimoGoebel

As stated before, I strongly believe these tests should go in to the core repo and run with each PR or it will be a pain to maintain this.
If we do it in a separate repository, I don’t ever want to see any core PR blocked because it breaks “smoker” tests. Separate repo means separate responsibility (read: somebody else’s problem). I know I’m probably not making a lot of friends with writing this, but I have seen in the past that the “definition of done” in core gets harder and harder to reach.

Will tests for every plugin live in this repo? How do you match the tests and for which foreman/plugin version they are valid?

@TimoGoebel From the RFC - Integration Tests & End to End tests - #3 by mmoll thread, it sounds like there are two efforts going on:

1. Create automated smoke tests that run on a release-basis (the ones @ekohl are proposing here)
2. Create automated UI smoke tests that run on a PR basis (the ones myself and @amirfefer have explored with cypress)

I think these can co-exist and serve different purposes. They also have different limitations, tests run on a release basis won’t have a huge time limit compared to tests run on each PR, which we would want to keep under the existing ~30 mins to run the tests (at least in Katello’s case). I agree that the PR level tests should live in the code itself. As a developer, if I break something in a PR, I don’t want to have to open up a separate PR to update tests, this is too hard to coordinate.

I think it’s fine if the per-release tests being proposed here live outside of the code, especially if they wont be plugin specific. Though we should do our best to keep these tests as flexible and forgiving as possible.

I’m glad to see testing efforts underway!