There's Plenty of Room at the Bottom: Analyzing and Verifying Machine Code
Thomas W. Reps and Junghee Lim and Aditya V. Thakur and Gogul Balakrishnan and Akash Lal
This paper discusses the obstacles that stand in the way of doing a good job of machine-code analysis. Compared with analysis of source code, the challenge is to drop all assumptions about having certain kinds of information available (variables, control-flow graph, call-graph, etc.) and also to address new kinds of behaviors (arithmetic on addresses, jumps to ``hidden'' instructions starting at positions that are out of registration with the instruction boundaries of a given reading of an instruction stream, self-modifying code, etc.).
The paper describes some of the challenges that arise when analyzing machine code, and what can be done about them. It also provides a rationale for some of the design decisions made in the machine-code-analysis tools that we have built over the past few years.
Paper available as: [PDF] [Official Version]