Fuzz Testing of Application Reliability
Classic fuzz testing is a simple technique for feeding random input to applications.
While random testing is a time-honored technique, our approach
has three characteristics that, when taken together,
makes it somewhat different from other approaches.
The input is random.
We do not use any model of program behavior, application type, or system
This is sometimes called
black box testing.
In the command-line studies (1990, 1995, 2006, and 2020), the random input was
random ASCII character streams.
For our X-Window study (1995), Windows NT study (2000), and Mac OS X study (2006),
the random input included cases that used structured input, valid keyboard and mouse
Our reliability criteria is simple: if the application crashes or hangs, it
is considerd to fail the test, otherwise it passes.
This criteria allows the use of a simple test oracle.
Note that the application does not have to respond in a sensible manner to the
input, and it can even quietly exit.
As a result of the first two characteristics, classic
fuzz testing can be automated to a high degree and results
can be compared across applications, operating
systems, and vendors.
We encourage your feedback and comments.
Below are links to the fuzz papers, software, and related materials:
1988 Original fuzz project assignment
Project (1) on the list is the fuzz assignment.
This is the original project assignment from my graduate CS736 class,
Advanced Operating Systems.
This list of projects was handed out in the Fall of 1988.
1990 Original Fuzz Report
B.P. Miller, L. Fredriksen, and B. So,
"An Empirical Study of the Reliability of UNIX Utilities",
Communications of the ACM 33,
12 (December 1990).
Also appears (in German translation) as
"Fatale Fehlertractigkeit: Eine Empirische Studie zur
Zuverlassigkeit von UNIX-Utilities", iX, March 1991.
This paper was award the 2022
Jean-Claude Laprie Award in Dependable Computing from the
IFIP WG 10.4 on Dependable Computing and Fault Tolerance.
1995 "Fuzz Revisited" Report
B.P. Miller, D. Koski, C.P. Lee, V. Maganty, R. Murthy,
A. Natarajan, and J. Steidl,
"Fuzz Revisited: A Re-examination of the Reliability of UNIX
Utilities and Services",
Computer Sciences Technical Report #1268,
University of Wisconsin-Madison, April 1995.
Appears (in German translation) as
"Empirische Studie zur Zuverlasskeit von UNIX-Utilities:
Nichts dazu Gerlernt", iX, September 1995.
2000 Windows NT Fuzz Report
J.E. Forrester and B.P. Miller, "An Empirical Study of the Robustness
of Windows NT Applications Using Random Testing",
4th USENIX Windows Systems Symposium,
Seattle, August 2000.
Appears (in German translation) as "Empirische Studie zur
Stabilitšt von NT-Anwendungen", iX, September 2000.
2006 Mac OS X Fuzz Report
B.P. Miller, G. Cooksey and F. Moore,
"An Empirical Study of the Robustness of MacOS Applications Using Random
First International Workshop on Random Testing,
Portland, Maine, July 2006.
2020 Fuzz Report
B.P. Miller, M. Zhang and E.R. Heymann,
"The Relevance of Classic Fuzz Testing:Have We Solved This One?",
IEEE Transactions on Software Engineering, accepted for publication,
The Classic Fuzz Software and Dataset Site
(Updated August 2020)
2008 Foreword for a book on fuzz testing
This is the prose for a foreword that I wrote for a book on fuzz testing.
It gives a nice summary of origins of the idea.
If you have reported on the use of the fuzz tools on
testing other systems or more recent testing of the systems that we have
send us email
and we would be glad to provide a link to the report:
Tue 17 May 2022 03:26:46 PM CDT