A concrete example of this is Foreman and the Ruby versions it supports. Foreman 3.8 supports Ruby 2.7, but at some point we want to support both 2.7 and 3.0. If you only store a config for foreman.git in jenkins-jobs then you miss something. You can keep a list of branches and map it, but essentially you’re duplicating a structure that already exists.
Today Foreman’s Jenkins configuration does not use this overrides (which can be implemented using Jenkinsfile in the repository).
Proposed solution
Within this RFC I’ve only considered Jenkins and GitHub Actions. The former because we already have an existing infrastructure and the latter because it’s already used in various places. There are other CI systems, but it would require a significant effort.
We could invest in our Jenkins infrastructure, but historically it has mostly been the infra team maintaining it. On the other hand, developers have adopted GitHub Actions by themselves. This is probably because testing out changes with GitHub Actions is easier. Because of that I’m proposing we standardize on it.
The Foreman project is already sponsored by GitHub which graciously provides us with a plan that includes more runners.
Note that when I say Foreman I really mean foreman.git, not the whole Foreman project.
Implementation
One design principle to keep in mind is that we should be writing as little as possible in GitHub workflows. If possible, logic should be in real files. That m
eans it may call a rake task, but inlining Ruby code in a workflow is not done. This keeps it easy to reproduce locally and less painful if a migration to anot
her CI system is needed.
Preparations have already started. Fixes #36913 - Set up GHA with matrix to run test on Ruby 2.7 by ofedoren · Pull Request #9900 · theforeman/foreman · GitHub added a basic CI pipeline that verifies Foreman. It has less coverage than our existing Jenkins pipeline (which also runs systems using a browser). So to consider it done, the workflow needs to be expanded.
It also uses a neat trick that @evgeni proposed. There is a matrix.json file in the repository which describes the test matrix. In itself this looks overly complex because you already have the whole workflow defined in the repository itself, but it’s aimed at plugins. A plugin’s workflow can use the same matrix.json file. The file has been backported to 3.8-stable and 3.7-stable, the currently su
pported versions.
This brings us to reuse. Actions by default are stored as a workflow inside a repository. If you want to have the same workflow in foreman_ansible and foreman_bootdisk then you need to copy it. Copying code (or configuration) is a bad practice because they can (and will) go out of sync. Luckily Actions can be reused. I’ve described the techniques that can be used, so I won’t duplicate it here.
There already is a workflow, but it’s incomplete and it’s not used everywhere. In Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub there is a investigation on the various differences. By now it may be outdated. Completing this will be a big benefit, since it already implements the aforementioned matrix.json.
postgresql-evr
Some plugins depend on postgresql-evr, which is a database extension to provide RPM epoch-version-release (EVR) logic. Realistically speaking this is only Katello, but there are plugins which depend on Katello.
Today postgresql-evr is installed as an RPM package to provide the extension and then activated, which means the postgresql container image can’t be used.
One alternative is to provide another container which includes the exention, but it may be better to create the extension in a migration. This is what pulp_rpm does. This has the additional benefit that using an external database because much easier, because only a vanilla PostgreSQL is required. Various clouds offer managed PostgreSQL and that could simplify operations for some users.
At this point either solution would be acceptable from this RFC’s point of view. It may also be an alternative container in the short term while working on the long term solution.
Concrete plan
There are some concrete actions which can be done in parallel:
- Expand Foreman’s CI GHA with missing steps
- Convert existing Foreman plugins to use the foreman_plugin GHA workflow (Foreman landscape provides an overview of plugins)
- Expand the foreman_plugin workflow with missing coverage (Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub mentions some)
- Find a solution for postgresql-evr
To convert existing Foreman plugins I’m proposing a collaboration. People involved in the system tech update project can start the effort and provide a good reusable action, but the various plugin maintainers should make sure their plugin takes advantage of it. This way we can work in parallel without overtasking some individuals.
I may have missed things. At worst, those would block the whole migration but more likely I’ve missed some steps.
)