Including selected tests from Foreman core during plugin test execution

Development RFCs
1

RFC: Including selected tests from Foreman core during plugin test execution

Decision Due Date: 2026-07-13

Context and Problem Statement

Foreman plugins today utilize the shared action from theforeman/actions to run tests. The main suite that gets executed there is rake test:$plugin_name which the plugin can define itself in its lib/tasks/$plugin_name.rake file, which in most cases is a "run all tests you can find in the test/ directory.

However, it can be beneficial to also run tests from Foreman core (e.g. access_permissions_test.rb).

We used to do this when we used Jenkins, but stopped with the migration to GHA.

Proposal

The “simplest” way to achieve this is to add the additional tests to theforeman/actions, e.g. as done in Run a predefined set of tests from core to ensure integration by evgeni · Pull Request #91 · theforeman/actions · GitHub

This has the downside that the definition which tests from core are valuable now lives inside the action and can’t be (easily) altered by neither Foreman core nor the plugins.

Alternative Designs

Explicitly adding the test into test:$plugin_name

This was proposed in Run permissions tests from Foreman core by ekohl · Pull Request #51 · theforeman/foreman_plugin_template · GitHub and some plugins do this already:
foreman_ansible/lib/tasks/foreman_ansible_tasks.rake at cba9c398aca724d467888cce93f2524e16952413 · theforeman/foreman_ansible · GitHub

However, this means that every plugin needs to opt-in to this method and while the original proposal by @ekohl is about 3 years old now, not many plugins have done so.

Extending test:$plugin_name from Foreman core

We could extend Foreman’s test.rake to have a dedicated target for “tests useful for plugins” and automatically add a dependency on this target to all Rake tasks that match test:foreman_ (that’s the only way to identify plugin tasks, sorry Katello and foreman-tasks).

The dependency injection seems hacky though :frowning:

Providing test:core_tests and letting the action call it.

Same as the “Extending” proposal above, but instead of injecting the dependency, we rely on the action to call it.

As the action is used by older Foreman releases too, this seems risky, as it would mean that we also need to backport test:core_tests to older releases.

Decision Outcome

Providing test:core_tests and letting the action call it. Fallback to rake test TEST="test/unit/foreman/access_permissions_test.rb" for Foreman versions that do not have that rake task. Plugins can opt-out.

Impacts

This should impact all plugins, by running a set of recommended tests from core during plugin testing.
The plugins that already do this themselves need to adjust (but running the tests twice is not really a problem either).

2

Thanks for the write-up @evgeni !

Personally, I think that all plugins should run the permissions test (or other tests from Foreman core if there are any), so I am fine with having it somewhere in the shared action. Would it be possible to add a flag to the action to make the test run optional if the flag is set? That would allow individual plugins to opt-out in case this really should be necessary.

3

Maybe we could make it an input, with the default value access_permissions_test.rb, then users can both opt-out (by setting it to an empty value) and adjust the list of tests if they want more? :thinking:

4

Are there that many additional tests in Foreman core that plugin maintainers would want to run?

If there are, passing a list of tests sounds like a good idea to me. I assume you’d have to pass the whole path of the file in that case.

5

I don’t know what is a good outcome because they all have trade offs. Instead, I’m going to add context in the hopes that it helps others.

And to expand on this: we actually ran the full Foreman test suite, not just some tests.

Back then it was way more common to monkey patch core and modify existing controllers that could break in some ways. It would be very hard to track what could break and also a lot of work to maintain.

The reason to keep control within the plugin is that that’s where you know what you’re patching.

So there is a variation on this: run the full test suite again. It will be expensive though, and take a long time.

Another variation is the middle ground: if test:core_tests doesn’t exist, can fall back to running the full test suite.

6

I really don’t like running the whole Foreman suite for every plugin PR, as I think the cost/value rate is rather low.

But this made me think: we could run the equivalent of if task_exists('test:core_tests') { rake test:core_tests } else { rake test TEST="test/unit/foreman/access_permissions_test.rb" }

I guess the real question becomes: who do we want to make in charge of the “list of tests to run” - core or the plugins?

7

Tests are a measure of quality assurance for a product. So if there is a test that makes sure there are less bugs in the whole ecosystem, I do not feel bad ‘forcing’ it on every part of this ecosystem. And if that particular test should be run for every plugin (optimally), it makes sense to have it somewhere in a central place. Having a set of standard tests from core with the possibility for plugins to opt out sounds like a fair solution to me.

8

That’s also a decent fallback.

Can core detect what is extended? Do we want to trust autodetection?

9

I am not aware of an easy way, no.

I’ve updated Run a predefined set of tests from core to ensure integration by evgeni · Pull Request #91 · theforeman/actions · GitHub with the latest version of what we have discussed here.

10

And here is the core part: Fixes #39545 - add test:core_tests for plugins to use by evgeni · Pull Request #11112 · theforeman/foreman · GitHub