RFC - Integration Tests & End to End tests

Development
17

I guess many of us have been interested in this :smile: Iā€™ve been testing out cypress for Katello. I have a vested interest in making sure the UI works properly as I have been moving our angular pages to webpack and would like to make sure the pages work as expected in webpack. Iā€™m hoping to include in automated testing in with the refactor, I think itā€™s a great opportunity to do so.

I created a PR here with some instructions on how to use it and a link to the video. Itā€™s fully functional locally but I havenā€™t added any CI implementation yet.

Like @amirfefer, I was very impressed with cypress.io. Itā€™s very simple to set up and provides a lot of tools that you can leverage for your testing strategy. Some things I like about it (sorry if this is repetative from what others have said about it):

  • It interacts with the application as a user would
  • You can easily record network requests as fixtures and stub them to keep any calls away from the backend server
  • Can run in headless mode or in a very helpful GUI.
  • In headless mode it records video
  • No coupling with the implementation, it can test rails/angular/react pages
  • No coupling with how our application is packaged, if it can resolve a UI from a hostname, it can test it out.
  • Very robust API with many tools that we could potentially use.

I was able to get it to communicate with the server, though Iā€™m not sure that is something we want for PR-level testing as that requires a full environment with our backend services. This is why I stubbed network requests. However, it does leave the possibility open for true e2e tests on a per-release or nightly basis

It also can be used for API testing, which is something we may want to explore.

The cons I could see are:

  • No firefox support (as has been mentioned)
  • Additional packages needed to support cypress
  • Since we donā€™t always have good UI tag selectors, you are sometimes forced to use a styling class to select a DOM element, which could be brittle (though I donā€™t think this is unique to cypress)
  • Stubbing fixtures could break with API response changes

As far as using for a Foreman plugin, the only part that I saw that may need to be shared is the login workflow. Because of the ā€œblack boxā€ nature of cypress, it doesnā€™t care if its working on a plugin/rails engine/whatever. This is good news in that plugins can use their own implementation, and not need to wait for changes from Foreman core. (at least as far as I can tell)

My goals are similar to the ones @Walden mentioned:

  • Have an automated UI ā€œsmoke testā€ that lives with the application so it can be updated when breaking changes are introduced.
  • Run on every PR and not take longer than the existing tests.

My proposal for Katello:

  • Introduce cypress.io testing as a PR-level job (travis or jenkins, Iā€™m not sure which is best yet) with this PR.
  • As part of moving Bastion pages to wepback, I would create a cypress ā€˜smoke testā€™ for that page that runs against our current page.
  • This test would be merged and run in automation against future PRs.
  • Then the page would be moved to webpack, where it should pass the same smoke test (as no functionality should be impacted).
3 Likes