Using JUnit in grading scripts

About this document

This document details my experiences using JUnit for grading CS 302 projects. I’m certainly no JUnit expert and this document won’t show you how to use every JUnit feature. However, it should give you enough background so that you can get started developing code to automate the most tedious parts of the grading process.
This is a beta document. Please contact me with feedback!

Download the code

You can download JUnit here; version 4.4 is the latest official release as of this writing. JUnit is constantly evolving, so you may want to check the site for the latest version. Note that JUnit from version 4 onward requires Java 1.5 features including annotations.

Your first test suite

Ad-hoc testing

Most of the testing we do as instructors specifies method inputs and expected outputs in an ad-hoc manner. This is fairly quick to set up, but can become unmanageable quickly as the test cases increase in complexity and number. In addition, there are a number of special cases we have to worry about: What if the code throws an exception? What if it is supposed to throw an exception, but doesn’t? We can deal with these cases (and others), but only by introducing a great deal of boilerplate code that is tedious and error-prone to develop.
Consider this test of the Math.sqrt() method. Even with only one test, there’s quite a lot of boilerplate. The test itself is contained in the if statement, as a condition that must not be false:

Math.abs((Math.sqrt(x) * Math.sqrt(x)) - x) > TOLERANCE

We set up the inputs for this test (assigning a value to x), we ran the test, and we checked the result. These steps are the same for every test. We can certainly cut and paste these, editing them for each test, but it would be nice if we could factor these out in a robust way — especially when we’re dealing with many cases, or when we’re dealing with methods that operate on objects and must maintain invariants.
If we discover that some test cases fail by throwing exceptions rather than by returning incorrect values, we’ll have to treat those individually, so a failed test won’t cause our entire suite to stop running — this is more boilerplate, of course.
We can do another kind of test to ensure that certain method invocations throw particular exceptions. Again, we must retype a lot of repetitive code for each such test to get the results that we want: an error message after the excepting statement (that shouldn't be reached if the exception is thrown), a catch block for the expected exception (so that the expected operation doesn't terminate the program), and a catch block that will handle all other exception types (to ensure that the code only throws the expected type of exception). Repeating code is tedious and error-prone, especially because we are liable to resort to a dangerous and uncouth technique: cut-and-paste.

Streamlining tests

JUnit enables us to streamline the testing process by eliminating boilerplate, automating test harness runs, and handling a variety of cases for us. The nice thing about JUnit is that it demands very little programmer overhead. As an example, consider the square-root test from above — we're going to convert it into a JUnit test class. You don't have to do anything more complex than importing some classes, putting each test case in a separate method, and annotating test case methods.
First, we import the relevant JUnit classes:

import org.junit.*;
import static org.junit.Assert.*;

Then, we put the test cases each in separate methods. It doesn't matter what we call these methods, but we do need to annotate each test case method with Test.

 @Test
 public void test1() {
     /* What we're testing in this method */
     final String TESTWHAT = "test 1:  sqrt(45) * sqrt(45) should be close to 45";
     
     double x = 45;
     assertFalse(TESTWHAT, Math.abs((Math.sqrt(x) * Math.sqrt(x)) - x) > TOLERANCE);
 }

When we run this test, it will print the string referenced by TESTWHAT if the assertion fails (that is, if the difference between sqrt(x) * sqrt(x) and x is greater than TOLERANCE). You can see our revised tester here. That's really all there is to it if we just want to get started. However, JUnit gives us some additional (very nice) capabilities. We'll talk about test fixtures and exception checking.

Test fixtures

We want our tests to each reflect a single kind of failure -- so they should be very small. (This isn't just good practice. Since JUnit will only report the first failure in a test case, it's essential if we want to identify all failures!) As a consequence, we may find ourselves setting up and tearing down some state or objects for each test. Sometimes, this is simple, such as when we'd like to execute each test on a newly-constructed object, to insulate ourselves from confounding interactions between tests. Sometimes, our setup is more complex: loading files, acquiring resources, populating databases. Fixtures are the setup and tear-down actions that execute before and after each test, and JUnit provides special support for making fixtures painless.
To execute some code before every test invocation, put it in a method annotated with Before. (Note that your tests can refer to instance fields of the tester class.) As an example, let's say that we wanted to allocate and populate an array of int values before each test on some code. We could simply define and annotate a set-up method:

import org.junit.*;
import static org.junit.Assert;

public class Fixtures {
    private int[] myData;

    @Before
    public void setup() {
        myData = new int[] {1, 2, 3, 4, 5};
    }

    @After
    public void cleanup() {
        myData = null;
    }

    /* Actual test methods go here. */
}

Notice that we've also declared a method annotated After — this will run after each test. You can declare as many Before and After methods as you want.

Exception checking

This section will appear soon. In the meantime, check the JUnit FAQ.

Some CS 302-specific tips

Template files

You can download an example test class as well as a test runner class that will execute all of the tests in one or many test classes.

Best practices

The same practices we encourage our 302 students to use while testing apply just as well to us: make tests small (with as few confounding variables as possible), repeatable, and thorough — they should cover most possible cases.
Be sure to develop positive, negative, and borderline cases.
Make your test method names self-documenting (perhaps even if this makes them comically long). You won't be invoking any of them explicitly (and thus typing them out), but the names will appear in failure reports.
Aid the graders by putting suggested point deductions in the detail messages for failed tests.

Where to go from here

Links

Be sure to read the JUnit FAQ, especially the section on best practices.
Popper extends JUnit with support for theories, which provide syntactic sugar for verifying the behavior of code over many data points. (It is not a theorem prover or model checker — it merely automates the process of finding a counterexample to an invariant over a large space of possible inputs.) I haven’t used it, but it looks interesting.

Author information

Please with any questions, comments or concerns about this document.