Hi,
there is an increasing number of plugins that have other plugins as a dependency. This creates a need to run tests against the checkout of latest master/develop branches for core + (at least) 2 plugins, which is currently not supported on Jenkins.
Some plugins have been dealing with this by moving tests to Github actions due to the following reasons:
-
modifying jobs on Jenkins is difficult to test
-
modifying jobs on Jenkins can potentially impact others as the job configuration is shared
-
job configuration is located in a separate repo and to change it means putting additional burden on members of the platform team who have to review/test/maintain it
However testing with gh actions has shortcomings as well:
-
when testing plugins on Jenkins, we currently run the whole test suite for core + plugin, which seems like a sensible approach given the way we extend core classes. Missing packages (such as libvirt) on gh action runners do not allow us to run the whole test suite for core
-
the tests for core take quite some time to finish and we would definitely hit the limits for the free tier
Traditionally, core has provided support for plugins. The question is: Will we continue or will we leave the testing setup on individual plugin maintainers?
There is a possibility to go for a paid tier of gh actions, with the option to migrate some of the machines currently used as Jenkins builders and turn them into gh action runners. Main benefit here would be putting job configuration under direct control of developers and reducing the amount of common configuration the platform team maintains. Further investigation of cost and how custom gh runners work is needed though.
Katello currently has a custom test job on Jenkins as noted by @evgeni.
@ehelms pointed out that this aligns with the effort of refactoring the existing test jobs and Jenkinsfile usage.
