Convert Foreman & plugins continuous integration to GitHub Actions

A concrete example of this is Foreman and the Ruby versions it supports. Foreman 3.8 supports Ruby 2.7, but at some point we want to support both 2.7 and 3.0. If you only store a config for foreman.git in jenkins-jobs then you miss something. You can keep a list of branches and map it, but essentially you’re duplicating a structure that already exists.

Today Foreman’s Jenkins configuration does not use this overrides (which can be implemented using Jenkinsfile in the repository).

Proposed solution

Within this RFC I’ve only considered Jenkins and GitHub Actions. The former because we already have an existing infrastructure and the latter because it’s already used in various places. There are other CI systems, but it would require a significant effort.

We could invest in our Jenkins infrastructure, but historically it has mostly been the infra team maintaining it. On the other hand, developers have adopted GitHub Actions by themselves. This is probably because testing out changes with GitHub Actions is easier. Because of that I’m proposing we standardize on it.

The Foreman project is already sponsored by GitHub which graciously provides us with a plan that includes more runners.

Note that when I say Foreman I really mean foreman.git, not the whole Foreman project.

Implementation

One design principle to keep in mind is that we should be writing as little as possible in GitHub workflows. If possible, logic should be in real files. That m
eans it may call a rake task, but inlining Ruby code in a workflow is not done. This keeps it easy to reproduce locally and less painful if a migration to anot
her CI system is needed.

Preparations have already started. Fixes #36913 - Set up GHA with matrix to run test on Ruby 2.7 by ofedoren · Pull Request #9900 · theforeman/foreman · GitHub added a basic CI pipeline that verifies Foreman. It has less coverage than our existing Jenkins pipeline (which also runs systems using a browser). So to consider it done, the workflow needs to be expanded.

It also uses a neat trick that @evgeni proposed. There is a matrix.json file in the repository which describes the test matrix. In itself this looks overly complex because you already have the whole workflow defined in the repository itself, but it’s aimed at plugins. A plugin’s workflow can use the same matrix.json file. The file has been backported to 3.8-stable and 3.7-stable, the currently su
pported versions.

This brings us to reuse. Actions by default are stored as a workflow inside a repository. If you want to have the same workflow in foreman_ansible and foreman_bootdisk then you need to copy it. Copying code (or configuration) is a bad practice because they can (and will) go out of sync. Luckily Actions can be reused. I’ve described the techniques that can be used, so I won’t duplicate it here.

There already is a workflow, but it’s incomplete and it’s not used everywhere. In Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub there is a investigation on the various differences. By now it may be outdated. Completing this will be a big benefit, since it already implements the aforementioned matrix.json.

postgresql-evr

Some plugins depend on postgresql-evr, which is a database extension to provide RPM epoch-version-release (EVR) logic. Realistically speaking this is only Katello, but there are plugins which depend on Katello.

Today postgresql-evr is installed as an RPM package to provide the extension and then activated, which means the postgresql container image can’t be used.

One alternative is to provide another container which includes the exention, but it may be better to create the extension in a migration. This is what pulp_rpm does. This has the additional benefit that using an external database because much easier, because only a vanilla PostgreSQL is required. Various clouds offer managed PostgreSQL and that could simplify operations for some users.

At this point either solution would be acceptable from this RFC’s point of view. It may also be an alternative container in the short term while working on the long term solution.

Concrete plan

There are some concrete actions which can be done in parallel:

To convert existing Foreman plugins I’m proposing a collaboration. People involved in the system tech update project can start the effort and provide a good reusable action, but the various plugin maintainers should make sure their plugin takes advantage of it. This way we can work in parallel without overtasking some individuals.

I may have missed things. At worst, those would block the whole migration but more likely I’ve missed some steps.

4 Likes

+1

Does it mean that we are moving from Jenkins to GHA completely, or will keep Jenkins running as well?

In the foreman_plugin action, I see that Foreman is installed and then only plugin tests are run. Do we want to run full foreman tests as well? Or maybe just a specific suite of critical stuff.

And another question, the database job runs db:migrate and db:seed, then install the plugin and run migrate/seed again.

My question is, is this behavior the same in the production setups, or do we first add all the plugins and then run db commands only once, with everything together?

2 Likes

We’ve been discussing running “important parts” in always execute access_permissions_test from Foreman core by evgeni · Pull Request #16 · theforeman/actions · GitHub and Run permissions tests from Foreman core by ekohl · Pull Request #51 · theforeman/foreman_plugin_template · GitHub
The TL;DR is “yes, we want that, but not yet 100% sure how”

Well, given a user can install a plain Foreman and then install any plugin (minus Katello…) via a later installer run, the current implementation seems correct (but then again, I am biased, I wrote it :smile_cat: )

At this moment we’ll keep Jenkins running. Converting everything at once will be a lot more work and something we can decide on in the future. Our packaging pipelines run on a schedule and in GitHub Actions only the default branch can have a schedule, so it could very well be less flexible than what we have today. So this RFC is scoped to only foreman.git and its plugins.

If we want to properly cover the real world, we should be testing both cases:

  • Install Foreman + plugin → db:migrate & db:seed
  • Install Foreman → db:migrate & db:seed → install plugin → db:migrate db:seed

Because both can happen on end user systems and DB migration orders can easily break things.

Ideally we would also cover upgrades:

  • Restore some old DB → install Foreman + plugin → db:migrate & db:seed
2 Likes

By now the action should have good coverage. While it’s not complete (most notably, always execute access_permissions_test from Foreman core by evgeni · Pull Request #16 · theforeman/actions · GitHub), I’d like to ask plugin maintainers to start migrating their plugins.

How to implement

The action is documented in the README and contains a copy-paste example:

Just modify the plugin name and submit that as a PR to your repository. For new workflows GitHub won’t automatically start them if they come from a fork, so I’d recommend pushing it to the original repository instead of a fork.

Please link Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub in the pull request itself so it’s tracked.

Update: Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub is seeing good progress. Many have been merged.

There are 2 cases where I think it’s better to retire the plugin:

Then foreman_hooks is also questionable. It should be replaced with foreman_webhooks, but I’m not sure we put in the effort to get users to migrate. Right now it doesn’t really have CI so I’m suggesting to ignore it.

It came up that new workflows in PRs aren’t always started. This happens when a PR comes from a fork and the workflow is not known in the original repository.

One workaround for this is to push the changes to a branch on the repository. Once the workflow is started, you can delete the original branch again.

What I always do is:

cd /path/to/checkout
hub pr checkout 1234
git push upstream HEAD

Then I wait for actions to start before I delete the branch branchname again.

# replace branchname
git push upstream :branchname

You will need push permissions to the upstream repository, so if you don’t then talk to a maintainer or GitHub admin. Asking on our Matrix dev channel with a link to the PR is also a good way.