Mutating Spring

One of the most requested features for pitest over the years has been support for mutating annotations. It’s easy to understand why. Popular frameworks such as Spring allow key functionality to be defined without writing a line of traditional code. If you’re working on a Spring codebase, having the right annotations in the right place is vital to correct functioning of the application.

Unfortunately, while it is quite straightforward to remove or alter annotations on a class, doing so in a way that is useful is much harder than it first appears.

Annotations aren’t code in the traditional sense. You can’t execute them. On their own, they don’t add any behaviour to a class for us to test. The behaviour is added by a framework that understands the annotation, so useful mutations can only be created if our mutation testing tools understands the frameworks that read them.

Or if the tool is really dumb.

The dumb approach that works

If your mutation testing tool is very simplistic, then annotations cause no issue at all. If the tool simply makes changes to source files then re runs the build to see if a test fails, this will work 100% of the time for any type of mutation. Unfortunately, the tool will be 98% useless, because the process is too slow to be practical. This is how the early open source tools used to work, and how many academic tools still work today.

The problem with being smart

In order to produce results in a reasonable timescale, pitest doesn’t recompile code, instead it manipulates bytecode in memory. And it doesn’t run all the tests against each mutant, it runs only the ones that execute the line of code the mutant is on. And it doesn’t launch a new process for each mutant, it recycles jvms.

This introduces a host of problems for annotation mutations.

• If the annotations are processed at build time, it won’t work at all
• If the annotations are processed at runtime, that processing may happen only once and be cached by the framework in the jvm
• The annotations are not on a executable line of code, so targeting mutants by line coverage won’t work

This is why annotation mutations have remained a much requested, but not implemented feature for so long.

A new plugin

Fortunately, thanks to our growing number of subscribers, we’ve been able to put time and thought into the problem. We’re pleased to announce we now have a preview release of spring support for pitest

The first release introduces a new mutator, but more importantly it hooks into the spring framework to ensure that state held by spring in the jvm is reset between tests. This is necessary to support annotations, but also solves problems for normal mutants within spring managed beans. Tests using spring extensions that were previously unable to kill mutants now can.

The first annotation mutator

For our first operator, we targeted validation annotations. These are widely used, and produce high quality mutants with no overlap with existing mutation operators.

The operator targets the following annotations, removing them from fields and method parameters.

• jakarta.validation.constraints.Min
• jakarta.validation.constraints.Max
• jakarta.validation.constraints.NotEmpty
• jakarta.validation.constraints.Pattern
• javax.validation.constraints.Min
• javax.validation.constraints.Max
• javax.validation.constraints.NotEmpty
• javax.validation.constraints.Pattern

These annotations are not tied to spring, but we cannot guarantee that mutation would be detected if the plugin were used with other frameworks.

Early feedback has been very positive. If you’d like to give the spring plugin a try, access is already included with our pro subscriptions. If you don’t have a subscription yet, you can email us for a free evaluation licence.

We hope to add more spring specific mutation operators soon. Two additional mutators have now been implemented, removing security and response annotations.