RFC - Integration Tests & End to End tests

My (selfish) intent is that it I want to verify a (remote) server installation with 0 interaction on the actual system. I want to verify every page at least loads without errors so I know that the packages we have in a repository are at least doing something for the end user. The aim is to verify our build system works.

Possibly, but I don’t intend this tool for developers. Its aim is automatically verifying the builds. The intended step for developers is to spin up a similar box and reproducing it should be trivial: log in and navigate to the page. Open your browser’s console log and see.

Agreed, but this was a solid base line. These URLs haven’t changed in years and users might actually have them bookmarked. They should be stable and breaking those URLs should be considered breaking the application.

Ideally there would be a JSON HTTP endpoint (so there’s no local interaction on the system) that returns all the menu items so you can also automatically verify all plugins.

No. As I said: this is solving my need (RFC - Integration Tests & End to End tests). Since there hasn’t been any progress in the past 6 months on an item I find extremely important in my daily job, I solved my own problem. Others are free to implement it in a way they can maintain and if it solves my need, I’m happy to drop my implementation. It was just half a day so I’m not attached to it.

The forklift project is already a lot of Ansible and Python so it’s not unusual there. QE also writes their tests in Python with pytest (@rplevka is working on end to end provisioning tests) so I’m also happy to move it there in the long run.

I thought we already have integration tests in the framework using Capybara. That can use selenium which in turn uses an actual browser. All the tooling to that exists to day. If it’s broken, we should fix it. It is written in Ruby. What would limit us from actually using this?

Again, this is not going to solve my personal use case.

Strongly disagree, at least for my goal. There should be no coupling and on the system under test and the tooling. Browsers that implement the webdriver API provide controls via a REST API. I’m only relying on what the browser is able to do to emulate an end user. Having to write Javascript would be a huge negative for me. Also because it means we would need javascript in our (production) test setup which we currently don’t.

IMO you should continue with your effort and merge it into froklift as this is the best solution we have atm and it gives us some great benefits.

While you do that, let’s continue to figure out what is ideal for foreman.



I would love to see us continue working with cypress and figuring out how we can set up tests on a PR level. Until then, lets use what works and makes sure our users don’t see blank pages :sweat_smile: