RFC - Integration Tests & End to End tests

[ekohl]

Thanks for doing this. I want to have this, but I do have an objection: The target repo. I want to have this in core and run with every PR. Why would you want to have this in a separate repo?

My (selfish) intent is that it I want to verify a (remote) server installation with 0 interaction on the actual system. I want to verify every page at least loads without errors so I know that the packages we have in a repository are at least doing something for the end user. The aim is to verify our build system works.

Can we run the tests in a Docker container so a developer does not need a special environment to run the tests in?

Possibly, but I don’t intend this tool for developers. Its aim is automatically verifying the builds. The intended step for developers is to spin up a similar box and reproducing it should be trivial: log in and navigate to the page. Open your browser’s console log and see.

Hardcoded files need maintainance. If we hardcode them, we should have tests in core that tell us that we missed a new route. But then again, we could just run a script to print all routes and feed your code with the script’s output.

Agreed, but this was a solid base line. These URLs haven’t changed in years and users might actually have them bookmarked. They should be stable and breaking those URLs should be considered breaking the application.

Ideally there would be a JSON HTTP endpoint (so there’s no local interaction on the system) that returns all the menu items so you can also automatically verify all plugins.

Is there a ruby or javascript implementation for this? Despite personal preferences, I’m a little reluctant on adding tests in yet another language.

No. As I said: this is solving my need (RFC - Integration Tests & End to End tests). Since there hasn’t been any progress in the past 6 months on an item I find extremely important in my daily job, I solved my own problem. Others are free to implement it in a way they can maintain and if it solves my need, I’m happy to drop my implementation. It was just half a day so I’m not attached to it.

The forklift project is already a lot of Ansible and Python so it’s not unusual there. QE also writes their tests in Python with pytest (@rplevka is working on end to end provisioning tests) so I’m also happy to move it there in the long run.

I would want something I can run for each PR so developers who made the error will have the tool to find, debug and fix the error before it gets merged.

I thought we already have integration tests in the framework using Capybara. That can use selenium which in turn uses an actual browser. All the tooling to that exists to day. If it’s broken, we should fix it. It is written in Ruby. What would limit us from actually using this?

Again, this is not going to solve my personal use case.

I would say javascript is ideal because it is mostly going to catch javascript errors.
Therefore using a javascript tool will make it easier to debug to javascript errors.

Strongly disagree, at least for my goal. There should be no coupling and on the system under test and the tooling. Browsers that implement the webdriver API provide controls via a REST API. I’m only relying on what the browser is able to do to emulate an end user. Having to write Javascript would be a huge negative for me. Also because it means we would need javascript in our (production) test setup which we currently don’t.

[sharvit]

My (selfish) intent is that it I want to verify a (remote) server installation with 0 interaction on the actual system. I want to verify every page at least loads without errors so I know that the packages we have in a repository are at least doing something for the end user. The aim is to verify our build system works.

IMO you should continue with your effort and merge it into froklift as this is the best solution we have atm and it gives us some great benefits.

While you do that, let’s continue to figure out what is ideal for foreman.

[John_Mitsch]

My (selfish) intent is that it I want to verify a (remote) server installation with 0 interaction on the actual system. I want to verify every page at least loads without errors so I know that the packages we have in a repository are at least doing something for the end user. The aim is to verify our build system works.

IMO you should continue with your effort and merge it into froklift as this is the best solution we have atm and it gives us some great benefits.

While you do that, let’s continue to figure out what is ideal for foreman.

+1

I would love to see us continue working with cypress and figuring out how we can set up tests on a PR level. Until then, lets use what works and makes sure our users don’t see blank pages