Molecule testing in foremanctl

Development RFCs
1

A POC by @Shimon_Shtein to introduce Molecule role-level testing for foremanctl installer.

This POC focused on two things:

  1. Theoretical part: how to organize tests and what is the scope for per-role tests.
  2. Practical part: how to connect the dots, and make sure molecule tests are enabled and working in the repo.

Theoretical part:

Problem definition

  • Currently foremanctl is tested by two options:
    1. The existing e2e pytest suite
    2. Robottelo high level scenario testing
  • Each role has a lot of parameters that change the behavior of the role. For example the certificates role has three possible operation modes:
    1. locally generated certificates
    2. Custom certificates
    3. foreman-installer inherited certificates
  • Roles rely on previous roles to be executed (for example foreman role will require a running database to spin up a proper foreman container)
  • Testing all the permutations for all the roles quickly becomes too hard.
  • Currently there is no way to get a quick answer to a question “did I break something with my latest change”, which is really important for agentic development.

Assumption:

I see the foremanctl flow as a chain of transformations to a base server that result in a configuration to that server in such a way that it has foreman (or proxy, but I’m leaving the proxy out of the scope right now) responding on the machine.
It can be represented by: <clean server> => T1 => T2 => ... => Tn => <a server with foreman configured on it>, where each Tn is an ansible role applied to the server.
Given this representation, we can extract invariants. Those invariants are the contract between different transformations (roles). These contracts are a set of artifacts on the server that are always produced by the role Tn and are consumed by T(n+1). In addition each transformation can change the contents of produced artifacts according to the input parameter values.

Permutations problem solution:

For each role we make sure that for any input, the role maintains the invariant.
For each input we validate that the output configuration changes according to the input parameters.
This solution confines each role in a set of boundaries, where as long as the invariants are preserved, the next transformation can happen. This reduces the complexity from <input variants for T1>*<input variants for T2>*...*<input variants for Tn> to <input variants for T1>+<input variants for T2>+...+<input variants for Tn>.

Quick answer solution:

While the e2e tests can validate that the server that is configured is fully functional, I suggest using mocking for testing individual roles. This will give some confidence that the transformation still confined in the invariant and the configuration changes according to the input parameters, but without the cost of running the full installation process.
Since we will use mock objects for containers instead of actual containers, we won’t need the actual prerequisites for the role, but on the other hand, we will still be able to inspect that the configuration objects that are expected from the role exist and contain the correct data.

The practical part:

Open discussion:

Is mocking the correct way to go forward.

Decision Outcome

Should we proceed with this effort?
Is it something useful?
Are there alternatives?

2 Likes
2

It fills the gap between unit tests and full E2E deployments by validating individual role behavior.
It is more benificial to Devs than to QEs
We can Incrementally and Selectively add it. However i would say we should not make molecule testing mandatory for every role untill we see maintenance and actual bug-catching effectiveness.
We should see it as a complementary dev focused testing layer, not a replacement for E2E or Robottelo testing.

3

In my opinion.
For Dev quick validation, Yes.
It will help devs and agentic workflows get quicker feedback before reaching full deployment testing.

4

Yes, the idea wasn’t to replace the E2E/Robottelo testing. I would say that we should have at least some e2e run before the actual merge.
This proposal adds an additional lower level of testing, as you have mentioned in your second comment.

5

Yes, It is very good addition specially for -

  • To validate a role change in minutes instead of waiting for a full deployment and E2E execution.
  • If a Molecule test fails, it’s usually clear which role is broken, making debugging easier.
  • It can be tested without creating dozens of E2E combinations.
  • It will catch issues with templates, file creation, permissions, and role logic that unit tests may miss.
6

My impression of molecule has always been that it’s meant as an integration test. Creating mocks will probably eat up a lot of time and is that really helping us?

My suggestion is to first start with testing individual roles and see how quick that is. Let’s not optimize prematurely.

1 Like
7

I too would prefer not to mock things. Molecule is great for integration tests on role level, and the integation IMHO includes that the deployed service is working properly.

1 Like
8

While writing this thread I realized that without mocks we will have trouble to fulfill the prerequisites: let’s say we want to test the foreman role (the one that installs the foreman container). If we want to test it with actual foreman container we need to make sure there is a database running and it’s configured to foreman’s “standards”. So individual role testing may become problematic.

I would say that the molecule testing in our case will verify that the installation process is working and all the invariants are met. We will be able to validate the viability of the installation later with full e2e suite.

9

So to summarize this discussion, I can see two main follow up questions:

  1. Should we switch from forge to molecule to manage our development scenario?
  2. Should we add a role-specific isolated testing? (Molecule is optional here, but IMHO it will make enough sense). The main difficulty with this approach is the fact the roles are very dependent on side effects of previous roles. For example the role that spins up a Foreman container depends on the role that deploys a database container. This will make integration (non-mocked) testing of the Foreman role quite hard.