Foreword for Fuzz Testing Book
Below is the draft of prose that I wrote for the foreword to a book
on fuzz testing:
It was a dark and stormy night. Really.
Sitting in my apartment in Madison in the Fall of 1988, there was a wild
midwest thunderstorm pouring rain and lighting up the late night sky.
That night, I was logged on to the Unix system in my office via a
dial-up phone line over a 1200 baud modem. With the heavy rain, there was
noise on the line and that noise was interfering with my ability to type
sensible commands to the shell and programs that I was running. It was
a race to type an input line before the noise overwhelmed the command.
This fighting with the noisy phone line was not surprising.
After all, this was just before error-correcting modems were available.
What did surprise
me was the fact that the noise seemed to be causing programs to crash. And
more surprising to me were the programs that were crashing -- common Unix
utilities that we all use everyday.
The scientist in me said that we need to make a systematic investigation
to try to understand the extent of the problem and the cause.
That semester, I was teaching the graduate Advanced Operating Systems
course at the University of Wisconsin. Each semester in this course,
we hand out a list of suggested topics for the students to explore for
their course project. I added this testing project to the list; the text
for this project description appears in the break-out box below.
In the process of writing the project description, I needed to give this
kind of testing a name. I wanted a name that would evoke the
feeling of random, unstructured data. After trying out several ideas,
I settled on the term fuzz.
Three groups attempted the fuzz project that semester and two failed to
achieve any crash results. Lars Fredriksen and Bryan So formed the
third group, and were more talented programmers and more careful
experimenters; they succeeded well beyond my expectations.
As we reported in the
they could crash or hang between 25-33% of the utility
programs on the seven Unix variants that they tested.
However, the fuzz testing project was more than a quick way to find program
failures. Finding the cause of each failure and categorizing these failures gave
the results deeper meaning and more lasting impact. The source code for the
tools and scripts, the raw test results, and the suggested bug fixes were
all made public. Trust and repeatability were crucial underlying principles
for this work.
In the following years, we repeated these tests on more and varied Unix
systems for a larger set of command-line utility programs and expanded our
testing to GUI programs based on the then-new X-window system
Windows followed several years later
and, most recently, MacOS
In each case, over the span of the
years, we found a lot of bugs and, in each case, we diagnosed those bugs
and published all of our results.
In our more recent research, as we have expanded to more GUI-based application
testing, we discovered that classic 1983 testing tool, "The Monkey"
used on the earlier Macintosh computers [cite Hertzfeld book]. Clearly a
group ahead of their time.
In the process of writing our early fuzz papers, we came across strong
resistance from the testing and software engineering community. The lack
of a formal model and methodology and undisciplined approach to testing
often offended experienced practitioners in the field. In fact, I still
frequently come across hostile attitudes to this "stone axes and
bear skins" (my apologies to Mr. Spock) approach to testing.
My response has always been simple: "We're just trying to find bugs".
As I have said many times, fuzz testing is not meant to supplant more systematic
testing. It is just one more tool, albeit an extremely easy one to use,
in the tester's toolkit.
As an aside, note that the fuzz testing has not ever been a funded research
effort for me; it is a research avocation rather than a vocation. All the
hard work has been done by a series of talented and motivated graduate
students as class projects in my Advanced Operating System class.
This is how we have fun.
Fuzz testing has grown into a major subfield of research and engineering,
with new results taking it far beyond our simple and initial work. As
reliability is the foundation of security, so has it become a crucial
tool in security evaluation of software. Thus, the topic of this book
is both timely and extremely important. Every practitioner who aspires to
write safe and secure software needs to add these techniques to their bag of
Operating System Utility Program Reliability - The Fuzz Generator
The goal of this project is to evaluate the robustness of various UNIX utility
programs, given an unpredictable input stream. This project has two parts.
First, you will build a "fuzz" generator. This is a program that will output
a random character stream. Second, you will take the fuzz generator and use
it to attack as many UNIX utilities as possible, with the goal of trying to
break them. For the utilities that break, you will try to determine what type
of input cause the break.
The fuzz generator will generate an output stream of random characters.
It will need several options to give you flexibility to test different
programs. Below is the start for a list of options for features that
fuzz will support. It is important when writing this program to use
good C and UNIX style, and good structure, as we hope to distribute
this program to others.
-p only the printable ASCII characters
-a all ASCII characters
-0 include the null (0 byte) character
-l generate random length lines (\\n terminated strings)
-f name record characters in file ``name''
-d nnn delay nnn seconds following each character
-r name replay characters in file ``name'' to output
The fuzz program should be used to test various UNIX utilities. These
utilities include programs like vi, mail, cc, make, sed, awk, sort,
etc. The goal is to first see if the program will break and second to
understand what type of input is responsible for the break.