Testing more plugins together regularly

Development RFCs
1

Motivation

As of today, we only run integration tests for Foreman and Foreman+Katello on a regular (daily) basis.
All other plugins are excluded and might be in a broken or even not-installable state.
It’s time to change this situation, at least for the major plugins we have.

Idea

Based on the pipelines we currently have for Katello (install and upgrade), we can design similar pipelines for other plugin sets. We can start with the “Luna” (see luna.yml) set and other sets can be added in the future when the current workflow is settled. As the current setup lacks tests for the various plugins, we will start with Katello tests and add plugin-specific tests later on.

Benefits

  • co-installability of plugin RPMs is tested
  • configuration (installer) and db-seeds of the plugins are tested in a production setup
  • upgrade issues can be caught early (even with an “empty” database)
  • allows us to add integration tests for the plugins

Drawbacks

  • will require compute resources (roughly the same as Katello), but this shouldn’t be a real problem these days
  • will require someone to look at and analyze the results of the runs

Possible implementation

We should start “small” and evolve from there.

Level 1

  • Clone Katello install and upgrade pipelines and make them install the Luna set of plugins
  • External proxy can be ignored if it’s too much work (but shouldn’t)
  • No new tests are written, only Foreman and Katello tests are executed.
    This would already ensure installability of the plugins plus proper upgrade paths.
  • Configure Jenkins to run the nighly version of these pipelines at least weekly

Level 1.5

  • When the pipelines are stable for a few runs, enable notifications to the “Infa & CI” group on discourse

Level 2

  • Add integration tests for the individual plugins

Level 3

  • Add pipeline definitions for released versions like it’s done for Katello

Level 4

  • Make the nightly pipeline block RPM promotion?

Proof of Concept

evgeni’s luna-pipe branch

7 Likes
2

Great! Sadly I have just one like that I can add to this post.

@Roman_Plevka builds something called smoke tests, that should run on any Foreman instance and is adaptive based on what plugin it can see. The idea is that it checks basic functionality in reasonable time. So far there’s an end to end provisioning test that finishes in 20 minutes. Would it be possible to have these running as part of this pipeline? It will live in robottelo but can be potentially extracted if I’m not mistaken. It needs few python packages to be present as well as libvirt-devel package. Given it’s testing provisioning on libvirt, in needs the host to have virtualization enabled, is that possible in our environment?

3

We run these pipelines on ci.centos.org, which provides us with beefy, virt-capable machines (no idea if that’s real metal or nested-virt, but we spin up VMs there ourself just fine). So yes, the environment would be able to run this.

And given provisioning is not a plugin as such, we can either add these tests as part of this effort (somewhere around L2) or individually even without this RFC.

1 Like
4

Thanks. These tests should hopefully also cover some plugins soon. Ideally the functionality we test during RC test weeks. I’ll be watching the progress, thanks a lot for your effort!

5

the level1 pr is up: forklift#928

6

level1 got merged now

7

level 1.5: https://github.com/theforeman/foreman-infra/pull/1021

2 Likes
8

and it failed already :slight_smile: luna-nightly-rpm-pipeline 5 failed