RFC - Integration Tests & End to End tests

ekohl · June 25, 2019, 6:07pm

Today I tried my hand at my immediate need: testing whether the pages actually load and no “white pages” are shown because webpack failed. My minimal implementation is:

gist.github.com

https://gist.github.com/ekohl/bcb4b815153f491d08d551ed4ed98ad9

defaults.json

{
  "username": "admin",
  "password": "changeme",
  "name": "Admin User"
}

instructions.sh

#!/bin/sh

# Tested on Fedora 30

# Install chromium and chromedriver
# Firefox hasn't implemented the webdriver logging - https://github.com/mozilla/geckodriver/issues/284
sudo dnf install chromedriver chromium

# Install python deps
python3 -m venv venv

This file has been truncated. show original

requirements.txt

pytest-base-url
pytest-selenium
pytest-variables
pytest-xdist

There are more than three files. show original

Sadly it’s limited to Chromium now because Firefox doesn’t implement the Webdriver logging so you can’t get the browser console log which my test relies on:

The generated html log (838.7 KB) contains the details for all the pages with their HTML source and a screenshot. By default it only takes a screenshot on failed tests but SELENIUM_CAPTURE_DEBUG=always can change that. Since it’s all pytest you can also add --junitxml=report.xml.

Note that I manually ensure the tests failed by inserting a console.error('Test') since no other pages logged any error (yay ).

On my laptop this takes about a 100 seconds.

If there are no objections, I intend to implement this in forklift so we can test our nightlies and solve my immediate need. After that I’m happy to accept improvements to trash it in favor of a more complete solution.

I’ll also look at differences between 1.20 and nightly. It should all work since the login process is the same and I’d expect URLs to remain the same. To support plugins (including katello) I’m wondering if I should extract URLs from the menu or use hardcoded files with paths to test.