Testing plugins using containers and GH actions

[Shimon_Shtein]

The problem

I am maintaining the rh_cloud plugin that has a hard dependency on Katello. The plugin uses GH Actions to test my PRs. Katello requires certain OS dependencies, like qpid e.t.c. The problem starts with the fact that GH Actions runs on ubuntu, but Katello dependencies are not available there.

Intermediate solution: Run containerized GH action.

This is a good solution that enables us to run our work on a centos-like environment. PR for reference: gh_actions to container by ShimShtein · Pull Request #639 · theforeman/foreman_rh_cloud · GitHub. The only thing that makes me uncomfortable with this solution is the time it takes to run: since we need to build the container from scratch, it takes ~20 minutes to run the tests from which ~15 minutes is the setup, and this is too much for my taste.

Proposed solution: build a custom container for GH action.

The idea is to use a prebuilt container instead of an empty centos7 container.
Advantages:

1. We can use the image as cache - we can prebuild as much as possible during the container build, leaving only the tests for the actual run phase.
2. We can run the tests locally exactly the same way they run on CI, so it would be much easier to debug CI failures
3. We can run the tests without installing heavy OS dependencies, for example if we use a distro that is not rh-derived.
4. We can run the tests using different versions of foreman and katello. In case of rh_cloud plugin, I have to maintain at least two versions of the plugin for each foreman version. Now it’s easy to run the tests against different foreman+katello combinations on the same machine.

Method:
I have added a Dockerfile that will be used to build the base container. We want it to start with an empty centos container, then add the missing packages: in my case I have added scl-ed git, postgres and node plus the dev dependencies needed for the tests to run. Next I have cloned foreman, katello and rh_cloud repos. And lastly, I have run bundle install and npm i to cache the packages.
Now at runtime we can bind the correct plugin code (the one that we actually want to test), rerun bundle install that would be quick, since the delta from the stable version is mostly small, rerun npm i to do the same for our js packages and finally run the tests.
PR for reference:

Use a prebuilt container for tests

One unresolved issue that I see, is the fact that npm i is not using local cache similar to bundler, so this step take longer than I want it to.

Suggestions, comments e.t.c. would be appreciated.

If you think this is a useful way, I would be glad to work on a more generic version of the container to accommodate also other plugins.

[Marek_Hulan]

I’d recommend talking with @Ondrej_Prazak he worked on automatic building of containers. I think there’s no progress yet with katello part, but nothing prevents us to continue now after it was migrated to pulp3. Also @ezr-ondrej may be interested in this thread.

[Ondrej_Prazak]

@Shimon_Shtein

If you think this is a useful way, I would be glad to work on a more generic version of the container to accommodate also other plugins.

I do find it useful. I previously attempted to build Foreman in container with a custom set of plugins in a generic way. My initial focus was more on the released Foreman versions, but additional goals further down the road were enhancements to CI and using images for tests fits into the whole picture quite well.

[lzap]

One concern is that we need to keep in mind GHA limits:

Usage limits, billing, and administration - GitHub Docs

They are currently decently sized for free tiers. But it can get out of control.

[ekohl]

I didn’t want to reply until it was certain, but this may help with GHA limits:

Our Github organization is now sponsored on the teams plan

Yesterday I reached out to Github and and happy to announce that we are now sponsored by Github. Our organization has been upgraded to the Teams plan (see https://github.com/pricing#compare-features and Usage limits, billing, and administration - GitHub Docs). Additional notes they sent me in the email: GitHub Discussions is now available, and you can convert Issues to Discussions and vice versa right on the repo GitHub Discussions is out of beta | The GitHub Blog Issue forms recently shippe…

[nadjaheitmann]

@Shimon_Shtein We maintain the scc manager plugin which also depends on Katello and switched to github actions CI, recently. We are mainly running unit tests, so we are fine on Ubuntu after setting up a custom postgres service container that supports the evr extension - at least for now. I think it would be really nice to have a common CI configuration for Katello plugins, especially for advanced testing

[m-bucher]

It might be enough to have ‘maintained’ container-images with foreman and maybe foreman-tasks and/or katello that are ‘only’ capable to run linting and unit-tests.
Whether they then run on github-actions, on my local machine, or my GitLab-Pipeline should not be that big a difference then.

One possible use case could be to run plugin-tests against various (supposed to be compatible) foreman-/katello-versions without the need to setup a build-environment first.