Hello everyone,
on this thread I would like to describe our concerns with capybara, especially from a Front-End developer perspective, and to suggest some alternatives.
Ideas and other thoughts will be more then welcome 
Present architecture
E2E
Pipeline bats, but not client edge - server edge running on a browser
Integration
foreman uses rails minitest with capybara (phantomJs driver)
While this integration test architecture has some benefits:
- Mocking data with
fixturesandfactoyBot - Direct access to models/DB
- Works naturally with rails
It defiantly has some cons that affect us:
-
Might be slow, casued by predefind timeouts
-
Debugging is difficult and frustrating, for example
- Poor ability to see why a test faild visually
- Having unclear JS errors
- Running a singular test might be encrounterd with errors (assets should be precompile, as long as webpack)
-
Random test failures
-
Abusing tests - some tests are written from a unit prespecitve
-
The current architecture seems to be incompatible with react, some tests are failing due to timeout errors, bad handling with promises, and async events (for instance, we have a workaround for AJAX requests
wait_for_ajax)
Alternatives:
- Gradually moving to another js driver - (we have couple of PRs for headless chrome i.e #21592),
- Replacing capybara for react pages with a different tool
- Gradually migrate capybara to a different tool
- Suggest any ?
On my searching for a modern testing tool, a tool named cypress took my attention the most
an open source project for full browser E2E testing
Pros:
- integrates with CI such as jenkins and travis
- recording tests - automatically record failing test with a video and a snapshot
- Optional dashboard
- Cypress could be used as a sass with a dashboard - grants an easy way to access recorded tests, fully integrates with travis.
This link will lead you to the dashboard of the demonstrated test I made.
-
great for debugging tests - Adding a local tool, watching a test on the browser live, plus
an ability to watch each step by demand. -
Clear API for writing tests
-
running tests concurrently
-
allows to execute processes - there are few gems that communicates with rails (cypress-on-rails)
Cons:
- No “out of the box” server side communication (besides 3rd-party gems)
- Different environment, javascript based
- ATM only chrome and electron are supported (supporting in firefox and IE are developed)
Demonstration
For the demonstration I’ve created a travis configuration with foreman test environment and cypress
Test example
test recording, recorded on travis environment
Test Code
import { FOREMAN_ADDR } from './config';
describe('Layout', () => {
beforeEach(() => {
cy.visit(FOREMAN_ADDR);
});
it('hover all menu items', () => {
cy.get('.fa-tachometer').trigger('mouseover');
cy.contains('Dashboard').should('be.visible');
cy.get('.fa-server').trigger('mouseover');
cy.contains('All Hosts').should('be.visible');
cy.get('.fa-wrench').trigger('mouseover');
cy.contains('Host Group').should('be.visible');
cy.get('.pficon-network').trigger('mouseover');
cy.contains('Smart Proxies').should('be.visible');
cy.get('.fa-cog').trigger('mouseover');
cy.contains('Users').should('be.visible');
});
it('taxonomies dropdown', () => {
cy.contains('Any Organization').click();
cy.contains('Manage Organizations').should('be.visible');
cy.contains('Any Location').click();
cy.contains('Manage Locations').should('be.visible');
});
it('notfication drawer', () => {
cy.get('#notifications_container').click();
cy.contains('No Notifications Available').should('be.visible');
});
it('user dropdwon', () => {
cy.get('#account_menu').click();
cy.contains('My Account').should('be.visible');
});
it('mobile view', () => {
cy.viewport(550, 750);
cy.contains('Toggle navigation').click();
cy.get('.visible-xs-block .fa-user').click();
cy.contains('My Account').should('be.visible');
});
});
As you can see, this test looks clear, no JS skills are really required, and with gems like cypress-on-rails it would
be pretty easy to migrate a test from capybara to cypress.
I will take any responsibility for such a gradually/partly migration, if decided.
Plugins
It’s a pain to have a broken plugin that affect the entire application.
Katello, for instance, doesn’t have any integration tests with foreman core, AFAIK.
With this tool plugins would be able to create integration tests more easily with almost the same travis configuration
and by that, plugins would be more stable with foreman core changes.
E2E
Beside integration tests, We can take these advantages, and create an E2E architecture for foreman (within production environment).
With it we can gain great benefits, for example, discovering js runtime errors, which happen after webpack compilation.
After all, this will increase foreman builds stability by far.
).
I’ve been testing out cypress for Katello. I have a vested interest in making sure the UI works properly as I have been moving