Hello everyone!
As some of you know, I’m pretty focused on quality. I also am a fan of tools that can be used to help improve quality. It is that reason that I’d like to propose the introduction of a mutation testing framework.
What is mutation testing?
Mutation testing essentially modifies the code in a project, in various ways, then runs that project’s unit tests. If a test still passes with mutated code, then that test is likely flawed in some way. An alternate form would be mutating the test assertions themselves.
Why use mutation testing?
It is cheap! You’ve all been busy writing unit tests for a long time, and those tests are extremely valuable. Mutation testing doesn’t need you to write a whole new suite of tests. Instead, it helps you to improve the tests already in place.
How would we use mutation testing?
Those decisions are largely up to people smarter than me. Implementation specifics will certainly need to be considered carefully, along with who will review the results and file issues for test fixes.
However, my initial thoughts are the following:
- Choose a mutation testing framework (mutant, mentat, etc)
- Create a CI process that will run the framework and provide the results.
(maybe on a per-pr basis) - Review the results and file issues for test fixes
- Fix the tests
- Enjoy a more stable Foreman!
As you are all aware, Foreman is a big project. There may be some unforeseen issues with a mutation framework handling a project of this scale. That is something that I’m honestly not sure about. However, I think the potential benefits of mutation testing warrant exploration.
I’m sure there will be more than a few questions, so I will do my best to answer. If, for some reason, I’m slow to respond on here, then feel free to ping me in #theforeman or #theforeman-dev on irc. My nick is jcalla.
Thanks for your time!
- Jake