Two weeks ago, we started to use ci.centos.org to begin testing nightly Katello builds. We would like to continue the trend of using ci.centos.org for more integration testing. A couple of the things that are planned to start testing are:
Upgrade tests, upgrade a katello box from the latest release to the nightly version. This would be included in the Katello nightly release job.
Foreman nightly release tests that will also use the ci.centos.org hardware.
Running bats tests for EL5 and EL6 clients to make sure there’s no failures with these.
Running integration tests with Foreman and other plugins and having nightly tests for these as well.
This is just what is planned for now. We want to include the community more with this though. What are other plugins/tests that you think would be helpful and add value. Hitting errors earlier is the goal for this. Any thoughts, questions and tips are appreciated!
We have already chatted about that we want to have ci tests with more plugins enabled for core prs. Similar to the overreached katello job we currently have. Is that something we can do? Maybe this can be an opt-in test for larger PRs.
Alternatively I’d suggest to replace the katello test with a test that just enabled our example plugin. That way we can make sure the plugin template has examples for common API endpoints and is actually a usable plugin. And I’d argue we’d catch more plugin breakages then with testing the katello plugin.
In addition a test that provisions a vm on VMware (our most used CR according to community surveys) would be awesome.
Whats the process for adding more jobs to this? I would like to float the idea for making use of it in https://github.com/theforeman/foreman-ansible-modules. Currently, we record API interactions - obviously using a real Foreman / Katello would be an improvement.
As shown in the last demo, we use pipelines. Currently only pipeline_katello_nightly.yml is used but a PR is open to convert Foreman. As long as you can write such a pipeline (essentially an ansible playbook) we can run it. In our Jenkins jobs there’s some generic code to pull out debug data and archive it.
There are 2 aspects to this we need to fix.
First of all we need automated tests. Writing a test to provision a host is not hard, but if it fails to provision, how do you get sufficient debug info so you can find out what happened? Sometimes there are transient failures (network issues to mirrors). Perhaps a a screenshot of the console can work.
Another difficulty is the need for a VMware license and I’m not sure we’d be able to access one in CentOS CI.
Note that we can still write the tests so we can manually kick them off during a test week. You’d be able to debug issues while there’s still a solid base line of what functionality works.
While not required, the first round of tests added make use of Forklift to do the heavy lifting and wrap it in patterns that CI needs. That is to say, if you write a Forklift playbook then running it becomes easy. The current tests @Zhunting has implemented in ci.centos.org come from here:
The playbooks in there spin up vagrant machines and then run a series of Ansible roles and tasks against them. If anyone can distill their test scenario down to a playbook like this then we can easily run it on ci.centos.org. Further, the benefit of capturing it this way is that anyone can run this same thing locally (assuming enough local resources on their machine) and test/debug it.
I think that it would be incredibly useful to be able to build rpms on demand on any PR we request it (through [build rpm] or similar). In case we wonder if a PR is going to break nightlies because of the packaging, we could try it very easily. Also there’s a number of people in our community who are testing features once they are packaged, and for them testing in a development environment is very cumbersome.
I’d start with unit tests first, but a full installation test could also make sense. E.g. generate rpms and setup a vm with these rpms.
I also know companies, that have a trigger in their PRs that deploys a full temporary demo environment. It would be awesome if we could have that. A PR would then create a “pr-1243-abc.demo.theforeman.org” box where you could login and do manual tests. That could also help UX people to review designs.
Yes, but that could also be helpful for the community to have a good example plugin with good test coverage.
to testing the template plugin too. Knowing that’s in working order is good for new developers.