Computer Sciences Department logo

CS 368-2 (2012 Spring) — Introduction to Automated Testing

It is always a good idea to test a piece of software before releasing it, or in classroom terms, before turning it in for grading. Testing of software is a complex topic, and not one that I can do justice to in a brief web page.

One kind of testing is particularly useful: automated testing. That is, one uses a computer program (e.g., a Python script) to test another computer program (e.g., another Python script). The basic idea is to write test code that expresses how the code-under-test should perform. Then, you periodically run the test script to see if anything has broken.

Trivial Example

Suppose we have a function that calculates the sum of two numbers:

def sum_two(a, b):
    return a + b

Now, suppose we want to make sure this function works correctly. We could do something like this:

print 'sum_two(42, 13) = %d' % (sum_two(42, 13))

Then, we run the script, and make sure the output is correct. But there are several disadvantages to this approach:

So instead, we want to write a function like this (the actual details will be described below):

def test_sum_two():
    assert(sum_two(42, 13) == 55)

Essentially, this function says, “test that sum_two(42, 13) returns 55, and if it does not, raise an exception.”. You can see the actual function call and comparison directly, and the assert() function simply raises an exception when passed False.

Then, of course, we need a way to run all of our test functions and tally the results.

Let’s see how this works for real in Python.

Python Unit Testing

This kind of testing — one function at a time — is called “unit testing” (because we are testing small units of code). Python comes with built-in support for creating and running unit tests.

Preparing the Main Script

Before we can test a Python module (i.e., a script), it must follow certain guidelines:

The goal here is to make your module a real module, in the sense that it can be safely imported into another script. And that script will be our test script, which comes next…

Writing the Test Script

The basic format for the test script is this:


import unittest


class TestYourModule(unittest.TestCase):

    def test_something(self):
        # Add test assertions here
        # As many as you like
        # But typically, no more than a handful

    def test_something_else(self):
        # Include as many test functions as you like.
        # Typically, these functions are organized by
        # the function being tested, or by the test case.

    def test_more(self):
        # etc.

    # ...

if __name__ == '__main__':

All of the text that is highlighted in yellow are identifiers that you may name as you wish.

Writing the Tests

Within each test function, as shown above, you write code that runs code in your module and declares (or “asserts”) the expected outcomes. In most cases, running code from your module means calling functions in the module being tested.

Assertions are written using a handful of functions defined in the standard unittest Python module. Note that, because tests are written as class functions, that each assertion function call begins with the self. prefix. Here is a list of common assertion functions:

Technically, all you need is the first function; all others can be written in terms of it. And also technically, the message argument is optional in each case. However, this is the message that is printed when a test fails, and so it is critical for understanding your test results; do not omit it.

So, for example, here are some concrete assertions:

self.assert_(sum_two(42, 13) == 55, 'Two positive nonzero integers')   # OK way to do this test
self.assertEqual(sum_two(42, 13), 55, 'Same as above but using assertEqual()')   # Best way to do this test
if sum_two(42, 13) != 55:'Same again, but using fail()')   # Horrible way to do this test
self.assert_(my_string.isupper(), 'Should be all uppercase')   # Good use of assert_()

A Complete (If Basic) Example

Here is a simple module that says hello to a user. Note how the critical function, which creates the final output string from user input, does not do the input or output by itself; this makes it testable.



def make_greeting(the_name):
    return 'Hello, ' + the_name + '!'

if __name__ == '__main__':
    user_input = raw_input('Please enter your name: ')
    print make_greeting(user_input)

And now, the testing script:



import unittest
import say_hello

class TestSayHello(unittest.TestCase):

    def test_make_greeting(self):
        self.assertEqual(say_hello.make_greeting('Tim'), 'Hello, Tim!', 'normal case')
        self.assertEqual(say_hello.make_greeting(''), 'Hello, !', 'empty string')

    def test_make_greeting_2(self):
        except TypeError, e:
  'None as argument')

if __name__ == '__main__':

Running itself works as expected:

Please enter your name: Tim
Hello, Tim!

Running runs the tests and produces much different output:

Ran 2 tests in 0.000s


Note that the tests did not run the “main” code in, which is good, because we do not want to require user interaction during the (automated) test run. Instead, it ran the test functions in the test script. Each “.” in the output corresponds to one test function being run. Then, following the dashed line, we get a summary of the entire test run.

Suppose I introduce a formatting bug in make_greeting():

return 'Hello, ' + the_name

Now, running the tests shows that there is a problem:

FAIL: test_make_greeting (__main__.TestSayHello)
Traceback (most recent call last):
  File "./", line 11, in test_make_greeting
    self.assertEqual(say_hello.make_greeting('Tim'), 'Hello, Tim!', 'normal case')
AssertionError: normal case

Ran 2 tests in 0.000s

FAILED (failures=1)

Note the “F” instead of the first “.”, and the long FAIL message below the line of equal signs. This information helps you identify which test failed and what went wrong.