As a community we should be ready to move over to another service. foreman-ansible-modules has already been moved to Github actions (by @evgeni) because we were hitting limits in the past. Another alternative is our Jenkins setup.
I’d like to open the discussion on how we should be testing our code.
For the Puppet modules I’ve started a PR to see about the conversion from Travis to Github Actions:
Thanks for pushing it forward, IMO it’s a good idea start experiencing Travis with Github Actions, looks like they are more reliable and has more options.
Does this apply for the Jenkins tests? As a plugin maintainer I’m very unhappy with the current state of these tests. I believe, that’s why folks move to TravisCI.
If we’d move all CI testing to Github Actions, I’d be in favour of that. Especially the plugin tests. I believe we can provide an example if that’s be helpful.
I think that the issue with our current Jenkins setup is more the implementation than the actual long term sustainability. Those jobs need love.
One thing that I like much more about Jenkins is that it can parse test output via junit.xml while in Travis and Github Actions you need to look through the logs. I had looked at CircleCI as well but that asks for a lot of permissions so I started with GHA.
We’re using Travis quite extensively in Pulp. We started using it before they were bought by Idera. Previously, we were using Jenkins but it seemed like we had to spend at least one FTE to maintain it, and also the Jenkins instance was inside Red Hat’s network and although the results were reported on each PR, community members couldn’t actually see what was being run.
I’m planning on looking into Github Actions more after our release this month and I’d be interested to know more about how using Github Actions goes.