This artifact provides the toolchain used in the paper and implements a small subset of the experimentation performed in the paper; the subset used is necessary for artifact submission as the experiment as performed for the paper required tens of thousands of hours of processor time (and as such was executed on a high-throughput computing platform).
A Dockerfile is provided to automate the
process of executing the experimental subset. Building the Docker
instance from the Dockerfile (e.g.,
docker build . from the artifact's root
sudo use often required) will automatically run the experiment script
and display the results.
The toolchain (mutgen and mutins) can be built from the code directory with make.
The experimental subset (which builds the toolchain, harvests a set of mutants, and performs a subset of the paper's experimentation) is performed by the mutrun.sh script. The experimental subset performed by this script is chosen to take roughly half an hour on typical desktop hardware.
We have noticed some unusual behavior with the test suites when using certain versions of gcc, especially gcc version 6. As we do not currently have the resources to exhaustively test the experiment across a wide variety of environments, we strongly suggest use of the provided Docker container.
This artifact is designed to provide both the tools developed for the paper and an example of the experiment.
Due to the sheer size of the experimentation performed for the paper, it is not feasible to have the artifact duplicate these; we provide a subset of both the corpus to harvest a set of mutants from, and a randomly selected subset of mutation operators to insert into the system under test.