Testing foremanctl in a Multi-Flavor Environment

Development RFCs
1

RFC: Testing foremanctl in a Multi-Flavor Environment

TL;DR — foremanctl is adding multiple deployment flavors (katello, foreman, foreman-proxy-content). Each deploys different features and services. This RFC proposes approches/solutions
for multi-flavor foremanctl.

Context and Problem Statement

foremanctl will support multiple deployment flavors (katello, foreman, foreman-proxy-content, etc.). Each flavor deploys a different set of features and services. We need a test selection strategy that answers one question: given a flavor deployment, which tests should run?

Current state

Today, foremanctl supports a single flavor (katello). Tests live under tests/ and use pytest with testinfra to SSH into deployed VMs and assert against real services.

We have feature-gated tests which use @pytest.mark.feature("name") markers. At collection time, conftest.py calls foremanctl features to discover which features are enabled and skips tests whose required features are not enabled.

What changes with multiple flavors

New flavors are coming — at minimum foreman (standalone, no content management) and foreman-proxy-content (standalone content proxy). Each flavor deploys a different combination of features and services:

Proposal

Here are the list of proposed solutions/approches:

Approach 1: Feature Markers (Extend Current System)

Keep @pytest.mark.feature() as the sole selection mechanism. Make foremanctl features and features.yaml flavor-aware.

What changes:

  • foremanctl features must only list features valid for the deployed flavor
  • features.yaml needs internal features like pulp and candlepin so tests can gate
  • foremanctl deploy --add-feature must reject features invalid for the current flavor

Strengths:

  • One marker type for everything — simple and consistent
  • foremanctl features already exists; extending it keeps test infra minimal

Limitations:

  • Flavor blind * Currently foremanctl features list features without considering flavor(flavor blind), so all features are listed for every flavor.

  • Features != services making pulp an internal feature is a workaround that blurs the line between user capabilities and deployment topology

  • No negative testing skipping != asserting absence

  • Same thing with httpd tests, in case of standalone proxy, we need to test pulp routes but for plain foreman those does not exist

  • While our test works on enabled features we need to make sure we must enable relevant features for a flavor. As enabling non relevant features would invoke tests marked with that feature as well.

    Ex: we should not be able to add —add-feature katello to standalone proxy flavor.

Approach 2: Runtime Service Discovery

SSH into the deployment before test collection, probe running containers/systemd units and select tests(probably via markers but not restricted to it) from that map to run.

Strengths:

  • Uses actual deployed state. Flavor-independent — if Pulp is up, Pulp tests run.
  • Solves the feature-vs-service mismatch directly.

Limitations:

  • SSH overhead
  • flaky probe logic that itself needs testing, could run into false positive scenarios
  • Doesn’t prevent deploying invalid features, similar to first approach

Approach 3: Flavor-Specific Directories + Feature Markers (Recommended)

Per-flavor test directories. conftest.py reads the deployed flavor and deselects non-matching directories. Feature markers continue to gate optional capabilities.

Proposed test structure

tests/
    common/                          # runs on EVERY flavor
        foreman_target_test.py
        playbooks_test.py
    flavors/
        katello/
            postgresql_test.py       # asserts pulp + candlepin DBs exist
            httpd_test.py
        foreman/
            postgresql_test.py       # asserts pulp/candlepin DBs do NOT exist
            httpd_test.py
        foreman-proxy-content/
            pulp_test.py             # mirror-mode specific assertions
            httpd_test.py
    feature/                         # independent features
        hammer/
        foreman-proxy/
        bmc/
        iop/
        remote-execution/

Strengths:

  • Directory path IS the selection for flavor specific tests
  • Negative testing is natural for flavor specfic tests — each flavor asserts what must be absent
  • Solves the Pulp problem — pulp tests exist in both katello/ and foreman-proxy-content/ flavor directories, each tailored to how that flavor uses Pulp
  • CI is straightforward — one workflow per flavor**

Limitations:

  • Some test duplication across flavors
  • foremanctl deploy must still enforce flavor constraints
  • No negative testing for features, only for flavor specific tests

Decision Outcome

Decide one of suggested solutions or suggest a better one that could be used and implmented in foremanctl

Impacts

This decision will have significant impact on how we will introduce new flavors and tests those in foremanctl

1 Like
2

Why will we need those internal features? Right now we have features that represent Pulp things, using a “meta” feature, e.g. content/rpm, content/container How do you see those being handled?

What about flavors that effectively are a combination of flavors? For example, I would expect Satellite to have a flavor that encompasses nearly all of Katello and nearly all of Foreman.

Generally, I like the approach for #3 as I think it makes understanding the test directory nice and clear. I just have a few questions about the details.

3

Why will we need those internal features?

As of today, deploy playbook deploys all the roles and only gate some like hammer and foreman-proxy on enabled_features. but in case of vanilla foreman and foreman-proxy-content flavors we don’t need to deploy some of those roles. for ex: foreman-proxy-content flavor will have pulp(in mirror mode) but not candlepin, and it will be easy if we somehow have pulp and candlepin as internal features which can be used to gate those roles.

- role: candlepin
      when: enabled_features | has_feature('candlepin')

Right now we have features that represent Pulp things, using a “meta” feature, e.g. content/rpm , content/container How do you see those being handled?

Yes, agree, for all these content/ features we can just reply on single pulp as internal feature which represents content feature is enabled in the deployment and then run all content related tests which are marked as @pytest.mark.feature('pulp').
if there is any exception then we can just put it under tests/flavors/{flavor}/ directory.

What about flavors that effectively are a combination of flavors? For example, I would expect Satellite to have a flavor that encompasses nearly all of Katello and nearly all of Foreman.

AFAIK, for Satellite we can rely on katello flavor which depoys both foreman and katello. and if there is any other custom flavor(which needs diffrent combination of features), then we can just create that flavor and use it as we are gating our deployment on features not on flavor specifically.

Also it raises a question of How flexible do we want our flavor’s to be? for ex: I have deployed foreman flavor and now i want to make it katello, can i just do foremanctl deploy --add-feature katello and expect it to work exactly same as katello flavor?

4

What I had in mind with the pytest test suite is that it shapes itself to the environment, whatever that may be. Then how that environment exists is for another process.

Today that process is GitHub Actions or the local developer’s setup. But I’ve had my eye on tmt a while. Using packit (which we already have) we can also integrate with testing farm. Implement tmt tests by ekohl · Pull Request #168 · theforeman/foremanctl · GitHub exists, but I ran into issues with multi-machine setups.

The core idea is that you define various environments you want to test in and can also refine those further. In each of those you run some tests, which is our pytest suite. So you can define a setup for each flavor or combinations of things and run tests. tmt also has some concept of upgrades.

I know this isn’t exactly what you were going after, but I wanted to share some context from previous investigations.

1 Like