Pitest History with GitHub Actions

We’re big believers in dogfooding. We use Arcmutate for all our development, so we experience the same niggles and issues as our customers.

Of course, it isn’t quite that simple. Our customers use a range of different development workflows based on competing platforms such as GitHub, Bitbucket and Azure. We can’t use all of them day to day. But as much as we can, we like to mimic our customers’ experience.

Recently, an issue landed in our support queue from a customer experiencing problems using Arcmutate’s pull request integration alongside pitest’s incremental analysis feature. Mutation analysis generally runs very quickly on our codebases, so we don’t have much need for incremental analysis, but to ensure we don’t miss any further issues with how Arcmutate interacts with history files, we decided to add it to our builds.

Background

Incremental Analysis

Incremental analysis is a cool pitest feature which speeds up mutation testing by using information from previous runs to optimise the current one. It works by storing mutation results, and hashes of the relevent classes and tests, in a machine readable file.

If very little changes in the codebase between runs, the speedup a history file gives can be dramatic. Pitest is potentially able to assign a status to every mutant without running any tests at all.

Our Workflow

All our development runs on GitHub using GitHub Actions for CI. Every change is made on a feature branch and merged by a PR.

Most feature branches are either

1. Merged within an hour of being created
2. Deleted after several hours of experimentation (i.e they were a spike)

We often don’t know if we’re creating a spike of mergable branch until we’ve been coding for a while. It depends on what we discover as we explore the problem we’re looking at.

We open a PR with a WIP status as soon as the branch is created, and push often. GitHub Actions workflows then update the PR with automated analysis results including (of course) mutation testing. So we get constant feedback as we work.

The shortest lived PRs receive a single push before being merged, but some will receive multiple commits.

We run pitest across all changed classes. Even on a large PR, this usually takes less than 60 seconds and most of this time is the time taken to build and run the test suite.

Setting Up Incremental Analysis Using GitHub Cache

Incremental analysis is designed to speedup the analysis of an entire codebase. In some ways it is a partial replacement for version control integration. If you store and share the history file produced from a main branch, pitest can use it to (effectively) only analyse the changed code in branches created off it.

Since we are already analyse only changed code within a PR, there is little point in using a history file generated from a main branch. The only mutants created will be ones that cannot be accelerated using history from the main branch.

Instead, we decided to create a new history file for each branch. This has no benefit if the branch receives only a single push, but for branches that received multiple commits the history file will accelerate the re-analysis of mutants generated in the first commit. The potential speedup is quite modest, as the analysis times are already low, but incremental analysis could do its part to fight the climate crisis by saving some CPU cycles.

This approach also fits nicely with how the GitHub cache works, as caches are scoped to the branch.

Our workflow looks like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
- uses: actions/cache/restore@v3
  id: history-cache-restore
  with:
    path: '**/history.txt'
    key: $-pitest-history-
- name: run pitest
  run: ./mvnw -Ppitest -DhistoryInputFile=history.txt -DhistoryOutputFile=history.txt -Dfeatures="+GIT(from[HEAD~1]), +exclusionscan, +gitci" test-compile
- name: update pull request
  run: ./mvnw -e -B -DrepoToken=$ pitest-github:github
- name: "Check history files exist"
  id: check_files
  uses: andstor/file-existence-action@v2
  with:
    files: "**/history.txt"
- uses: actions/cache/save@v3
  id: history-cache-save
  if: steps.check_files.outputs.files_exists == 'true'
  with:
    path: '**/history.txt'
    key: $-pitest-history-$

The history-cache-restore step uses actions/cache/restore to look for any existing caches for this branch. If it finds one, it will be restored.

The run pitest step runs pitest, specifying that a file named history.txt should be used to both read and write history. It is important to understand that this history file is per module. All our builds are multi module builds, so there will be multiple history files produced. To make sure we didn’t check any of these in accidentally, we added a

1
*/history.txt

Entry to out .gitignore.

Once the analysis is complete (and the results have been updated in the PR) the cache is saved. If you try to overwrite an existing cache, GitHub actions will refuse with a warning, so we use the hashFiles function to append a hash based on the contents of the history files. The cache will therefore be written under a different key/name. This arrangement works because the cache restore action understands that the '-' is a delimiter. It will retrieve the most recently written key that matches before it.

The check_files step isn’t strictly necessary. We included it to prevent a warning from GitHub when we pushed changes to a branch that include only non Java files (such as documentation).

Using GitHub Cache Without Arcmutate

If you’re not an Arcmutate customer, you could still use a setup similar to the one shown above to reduce PR execution times. Although GitHub caches are scoped by branch, it is possible to read a cache from the base branch a feature branch was forked off. A history file could therefore be cached on a main branch, and read on the first run in any PR.

This approach would reduce execution times within the PRs, but a way would still need to be found to make the results of the analysis easily accessible. You’d also continue to miss out on all the other features Arcmutate has to offer.