Little Things

We’ve put some big headline features into Arcmutate: Kotlin support, test acceleration and pull request integration, but we’ve also added a lot of smaller things. They’re easy to overlook, but make mutation testing a much smoother process. Especially when you put them all together.

As they’re hard to describe in headlines alone, we thought we’d take a look at a few of them here.

Better Descriptions

Pitest mutates bytecode rather than source code. This makes generating mutants lightning fast, but it’s hard to stop bytecode details leaking out to the end user.

For example, if you feed this code to pitest

1
2
3
4
5
6
  boolean foo(List<String> list) {
    return list.stream()
      .filter(s -> s.startsWith("dog"))
      .findAny()
      .isPresent();
  }

It will produce a mutant with a description like

1
replaced boolean return with true for com/example/SomeClass::lambda$foo$1

This is a 100% accurate description of the mutant, but not very helpful unless you already understand the internal details of how lambdas are implemented in Java.

Arcmutate makes things a bit easier to understand, by describing the mutant like this

1
replaced boolean return with true for 1st lambda in foo

Byte code details are translated in other scenarios too, so anyone can read the descriptions without having to lookup sometimes obscure details of how Java is compiled.

Stable Mutants & Subsumption

It might seem obvious that generating (and killing) more mutants would give you more confidence that your code and tests work as expected, but the real picture is much more complicated. Additional mutants are only useful if they require different tests to kill them than the mutants you already have. If this is not the case, more mutants just means more processing time and more results that a human must look at and understand.

The best mutants are the ones which are harder to kill. These are said to be more stable.

For example, any test that referenced the return value of this mutation would kill it, even if it did no more than confirm it was not equal to null.

1
2
3
4
io.reactivex.rxjava3.core.Maybe<String> mutateMe(String a) {
    //return Maybe.just(a);
    return null; // mutant
}

The same test might not kill this mutation

1
2
3
4
io.reactivex.rxjava3.core.Maybe<String> mutateMe(String a) {
    //return Maybe.just(a);
    return Maybe.empty(); // mutant
}

It requires a test that at least confirms the Maybe holds a value. We could generate both mutations, but the null return one would not provide any value the empty one does not. It would just be extra noise.

Arcmutate can generate more stable mutants than pitest can in code that uses libraries such as Guava, RxJava and Project Reactor, and when it does it removes any less stable mutants that pitest generates. This is called a subsumption analysis, identifying mutants which overlap in the set of tests required to kill them, and eliminating ones which do not require unique tests.

Arcmutate can do it in other scenarios as well.

Given the code

1
2
3
boolean mutateMe(int a) { 
   return a != 42;
}

Pitest’s RETURN_VALS mutator will generate a mutant which makes the method always returns true, and the REMOVE_CONDITIONALS mutator will generate a mutant which removes the conditional statement, so again the method always returns true.

In a more complex scenario these two operators might result in code which behaves differently, but in this case the two mutants have exactly the same behaviour. Both mutants are equally stable, but there’s no point in generating both of them. Arcmutate’s subsumption analysis will detect this, and remove one of the duplicate mutants.

The end result is fewer mutants, faster processing times and fewer results for a human to interpret.

Conclusion

These are just some of the small improvements that Arcmutate offers, all included in the Base subscription. If you’d like to learn about other functionality included in the base package such as exclusions and extended mutation operators for modern Java code, take a look at the docs.