\section{Evaluation}
\label{sec:evaluation}

We implemented device state checkpoint and fine-grained fault tolerance
for the Linux 2.6.29 kernel for six drivers across three buses. The evaluation 
examines the following aspects of \toolshort:

\begin{enumerate}
\item {\em Fault Resilience.} What failures can fine-grained fault
  tolerance handle?  We evaluate \toolshort\ using a
  series of fault injection experiments and report our results.
% 
\item {\em Performance.} What is the performance loss  of fine-grained
  fault tolerance on steady-state operation? We report the performance cost for
  applying \toolshort\ on support  and  core I/O routines.
%
\item {\em Recovery Time.} What is the downtime caused by a driver
  failure? We compare the time taken by \toolshort\ to restore the device and
  cleanup the failed driver thread state with the time taken to unload
  and reload a driver.
%
\item{\em Usefulness of \toolshort.} Is selectively isolating entry
  points useful?  We evaluate whether suspect entry points can be
  identified in drivers and whether they reduce the amount of code
  isolated.
%
\item{\em Device Checkpoint Support.}  Is re-using existing power
  management functionality reasonable?  We examine the frequency of
  power management support in existing drivers that facilitates
  device checkpoints.
%  
\item {\em Developer Effort.} What is the overhead to the developer to
  enforce isolation in the system? We measure the effort needed to
  annotate a driver for isolation and to add checkpointing code.

\end{enumerate}
Unless otherwise specified, we compare \toolshort\ against unmodified drivers running on an unmodified 2.6.29 Linux kernel.



\subsection{Fault Resilience}

\begin{table}[t!]
\center
\footnotesize
\vspace{0.2cm}
\begin{tabular} {|p{0.9in}|p{1.9in}| } \hline

{\bf Fault Type} & {\bf Description of fault}
\tabularnewline  \hline

Corrupt pointers & Dynamic: Corrupt all pointers referenced in a
function to random values.
\tabularnewline \hline

Corrupt stack & Dynamic: corrupt execution stack by copying large
chunks of data over stack variable addresses.
\tabularnewline \hline

Corrupt expressions & Static: corrupt arithmetic instructions by
adding invalid operations (like divide by zero).
\tabularnewline \hline

Skip assignment & Static: remove assignment operations in a function.
\tabularnewline \hline

Skip parameters & Dynamic: zero incoming parameters in a function.
\tabularnewline \hline
\end{tabular}

\caption{\small \bf Faults injected to test failure resilience
  represent runtime and programming errors.  Dynamic faults are
  invoked using \codesm{\tt ioctls}, and static faults are 
  inserted with an additional compiler pass.}
\label{table:faulttypes}
\vspace{-0.27in}
\end{table}


% The purpose of this is to have centered column headings
% with right-justified numbers in the tables.
\newcolumntype{R}{>{\raggedleft} m{0.4in}} % Right justified
\newcolumntype{C}{>{\centering} m{0.4in}}
\newcolumntype{L}{>{\raggedright} m{0.4in}} % Left justified

\begin{table}[t!]
\center
\footnotesize
\centering
\vspace{0.2cm}

\begin{tabular}{|L||R|R|R|C|}
\hline
\centering {\bf Driver} &
\centering {\vfill \bf Injected Faults} &
\centering {\bf Benign Faults} &
\centering {\bf Native crashes} &
\centering {\bf \toolshort\ crashes}
\tabularnewline  \hline
8139too       & 43 & 0 & 43 & NONE \tabularnewline \hline
e1000         & 47 & 0 & 47 & NONE \tabularnewline \hline
ens1371       & 36 & 0 & 36 & NONE \tabularnewline \hline
pegasus       & 34 & 1 & 33 & NONE \tabularnewline \hline        
psmouse       & 22 & 1 & 21 & NONE \tabularnewline \hline
r8169         & 46 & 0 & 46 & NONE \tabularnewline \hline 
{\bf Total}   & {\bf 258} & {\bf 2} & {\bf 256} & {\bf NONE} \tabularnewline \hline
\end{tabular}
\caption{\small \bf Fault injection table with number of unique faults
  injected per driver.  \toolshort\ is able to correctly restore the
  driver and device state in each case.}
\label{table:injectionresults}
\vspace{-0.22in}
\end{table}

We first evaluate how well \toolshort\ can handle driver bugs
using a combination of dynamic and static fault injection over six
drivers. These tests evaluate both the ability of fine-grained fault
tolerance to isolate and recover driver state as well as the ability
of device state checkpointing to restore device functionality.
Table~\ref{table:faulttypes} describes the types of faults inserted in 
the SFI module. Static fault injection modifies the driver source code to emulate
programming bugs, while dynamic fault injection modifies driver data
while running to emulate run time errors. We perform
a sequence of trials that test each fault site separately.

During each experiment, we run applications that use the driver to
detect whether a driver failure causes the application to fail. For
network, we use \codesm{ssh}, and \codesm{netperf}, and for sound we use
\codesm{aplay} and \codesm{arecord} from the ALSA suite. We tested the
mouse by scrolling the device manually as we performed the fault
injection experiments.  After each injection experiment, we determine
if there is an OS/driver crash or the application malfunctions. We
re-invoke the failed entry point without the fault to ensure that it
continues to work, and that resources such as locks have been
released.

We injected a total of 258 unique faults in the native and \toolshort\
drivers. Table~\ref{table:injectionresults} shows the number of faults
injected for every driver and the outcome. For the native driver, all
but two faults crashed the driver or resulted in kernel panics. The
two benign faults were missing assignments.

In contrast, when we injected faults into driver entry points
protected by \toolshort, the driver and the OS remain available for every
single fault. Furthermore, in every case, applications continue to
execute correctly following the fault. For example, the sound
application \codesm{aplay} skips for a few milliseconds during driver
recovery but continued to play normally.  The shell notes this disruption
with the message ``\codesm{ALSA buffer underrun}.''

We also verify that internal driver and device state is correctly
recovered using the \codesm{ethtool} interface for network drivers. We
find that when failures happen while changing configuration
settings, re-reading settings after a crash always returns correct
values.

Finally, we verify that changes to drivers made using non-class
interfaces, such as the {\tt proc} and {\tt sys} file systems, before
injecting failures and present following recovery.  In contrast, shadow
drivers cannot replay these actions since they cannot capture
non-class driver interactions.



%%%
\subsection{Performance}

\noindent 
The primary cost of using \toolshort\ is the time spent copying data in
and out of the SFI module and creating device checkpoints. We measure
performance with a gigabit Ethernet driver, as it may send or
receive receive more than 75,000 packets per second. Thus, the
overhead of \toolshort\ will show up more clearly than on low-bandwidth
devices. We evaluate the runtime costs of using \toolshort\ and regular versions of
drivers in six configurations:
\begin{enumerate}
\item {\em Native:} Unmodified e1000 driver.
\item {\em \toolshort\ \subscript{static}:} Statically isolate 75\% of code
  (all off I/O-path). 
%\item {\em \toolshort\ \subscript{dyn}:} Dynamically isolate off-I/O-path code.
\item {\em \toolshort\ \subscript{dyn-1/2}:} Dynamically isolate every other packet in I/O path.
%\item {\em \toolshort\ \subscript{1/10}:}  Isolate I/O path code for every
%  10th packet.
\item {\em \toolshort\ \subscript{dyn-all}:} Dynamically isolate every packet in I/O path.
\end{enumerate}
The {\em dynamic} experiment measures the additional cost of choosing
at runtime whether to invoke the regular or SFI version of a
function. Finally, the {\em dyn-all} test represents the worst case of
invoking the SFI module on the I/O path for a high-bandwidth device.

Our test machine consists of a 2.5 GHz Intel Core 2 Quad system conﬁgured with 4 GB
DDR2 DRAM and an Intel 82541PI gigabit NIC running FGFT. We measure performance with
netperf~\cite{jones95} by connecting our test machine to another machine with 1.2 GHz
Intel Celeron processor and a Belkin NIC with a crossover cable.
Table~\ref{table:overhead} reports the average of 5 runs.

In the {\em static} test where code off the
I/O-path code is isolated, performance and CPU utilization are
unaffected. These results demonstrate that \toolshort\ achieves the goal of
only imposing a cost on isolated code. 

For the {\em dyn-1/2} test that isolates at runtime, entry points
on the I/O path (the packet send routine) for every other packet,
bandwidth dropped 4\% and CPU utilization increased 0.5\%.
Thus, selectively applying isolation, even on critical I/O paths, has
a small impact.

The performance drops further when we isolate critical path
code on every request since we copy shared driver and kernel data
across modules for {\it each} packet being transmitted. We find a 
7.5\% performance drop and 1\% higher CPU utilization.
\toolshort\ is designed to limit isolation costs to
specific requests and hence pays a cost of isolation because it
needs to setup isolation (create copies) as each packet requests
isolation. Overall, these results show that \toolshort\ can be applied 
at no cost to high bandwidth devices off the I/O path and at marginal cost
on the I/O path.

{\bf \em Device Locking}: We run netperf using multiple threads to measure 
the overhead from device locks introduced at isolated entry points.
We find that bandwidth drops an additional 2.6\% and 4.6\% with 10 and 100 
threads respectively, as compared to FGFT running in a single thread.  
We also test the {\tt \small ens1371} sound driver
and find that playing multiple overlapping sound files does not 
result in any lags or distortions. These results show that \toolshort\ introduces
low overhead with high bandwidth devices running multiple threads.  

\newcolumntype{R}{>{\raggedleft} m{0.6in}} % right justified
\newcolumntype{S}{>{\raggedleft} m{0.4in}} % right justified
\newcolumntype{C}{>{\centering} m{0.6in}}
\newcolumntype{L}{>{\raggedright} m{0.8in}} % left justified

\begin{table}[tbp]
\center
\footnotesize
{\bf Single Threaded}\tabularnewline  

\begin{tabular}{|L|R|S|} \hline

\centering {\bf System} &
\centering {\bf Throughput} &
\centering {\bf CPU}
\tabularnewline  \hline

Native & 843.6 Mb/s & 2.5\% \tabularnewline \hline
\toolshort\ \subscript{static}    & 843.6 Mb/s &  2.5\% \tabularnewline \hline
%\toolshort\ \subscript{dyn}   & 843.1 Mb/s &  2.5\% \tabularnewline \hline
\toolshort\ \subscript{dyn-1/2}      & 811.5 Mb/s &  2.9\% \tabularnewline \hline
\toolshort\ \subscript{dyn-all}	  & 784.4 Mb/s & 3.4\% \tabularnewline \hline
\end{tabular}

{\bf Multiple Threads}\tabularnewline  

\begin{tabular}{|C|L|R|S|} \hline

\centering {\bf Threads} &
\centering {\bf System} &
\centering {\bf Throughput} &
\centering {\bf CPU}
\tabularnewline  \hline

\multirow{2}{*}{10} & Native & 847.7 Mb/s & 3.0\% \tabularnewline 
& \toolshort\ \subscript{dyn-all}	  & 791.4 Mb/s & 4.2\% \tabularnewline \hline

\multirow{2}{*}{100} & Native & 941.7 Mb/s & 4.2\% \tabularnewline 
& \toolshort\ \subscript{dyn-all}	  & 860.2 Mb/s & 10.8\% \tabularnewline \hline

\end{tabular}

\caption{\small \bf TCP streaming send performance with netperf for
  regular and \toolshort\ drivers with checkpoint based recovery.}
\label{table:overhead}
\vspace{-0.2in}
\end{table}




\subsection{Recovery Time}
A major benefit of checkpoint-based recovery is the speed of restoring
service.  Table \ref{table:recovery time} lists the time taken by the
driver to recover using \toolshort\ and  by unloading and
reloading the driver. We measure recovery times by recording the time
from detection of failure to completion of the restore
routine. Overall, \toolshort\ is between 1.6 and 4,400 times faster
than restart recovery, and between 145ms and 1.5s faster. The drivers
with the largest speedup have complicated probe routines that are
avoided by restoring from a checkpoint. Hence, \toolshort\ provides
low-latency recovery and frequently offers an order-of-magnitude lower
recovery latencies.


\newcolumntype{R}{>{\raggedleft} m{0.42in}} % right justified
\newcolumntype{S}{>{\raggedleft} m{0.35in}} % right justified
\newcolumntype{C}{>{\centering} m{0.42in}}
\newcolumntype{L}{>{\raggedright} m{0.42in}} % left justified
\newcolumntype{M}{>{\raggedright} m{0.35in}} % left justified

\begin{table}[tbp]
\center
\footnotesize

\begin{tabular}{|M||M|M|R|R|S|} \hline
\centering {\bf Driver} &
\centering {\bf Class} &
\centering {\bf Bus} &
\centering {\bf Restart recovery} &
\centering {\bf \toolshort\ recovery} &
\centering {\bf Speedup}
\tabularnewline  \hline
8139too  &  net   & PCI   & 0.31s & 70$\mu$s  &  4400     \tabularnewline \hline
e1000    &  net   & PCI   & 1.80s &  295ms    &     6\tabularnewline \hline
r8169    &  net   & PCI   & 0.12s & 40$\mu$s  &  3000\tabularnewline \hline
pegasus  &  net   & USB   & 0.15s &    5ms    &    30        \tabularnewline \hline
ens1371  &  sound & PCI   & 1.03s &  115ms    &     9 \tabularnewline \hline
psmouse  &  input & serio & 0.68s &  410ms    &     1.65 \tabularnewline \hline

\end{tabular}
\caption{\small \bf \toolshort\ and restart based recovery times.
Restart-based recovery requires additional time to replay logs running over the 
lifetime of the driver. \toolshort\ does not affect other threads in the driver.} 
\label{table:recovery time}
\vspace{-0.2in}
\end{table}


\paragraph{Checkpoint/restore latency.}

\newcolumntype{R}{>{\raggedleft} m{0.5in}} % right justified
\newcolumntype{C}{>{\centering} m{0.5in}}
\newcolumntype{L}{>{\raggedright} m{0.35in}} % left justified

\begin{table}[tbp]
\center
\footnotesize

\begin{tabular}{|L||L|L|R|R|}
\hline
\centering {\bf Driver} &
\centering {\bf Class} &
\centering {\bf Bus} &
\centering {\bf Checkpoint times} &
\centering {\bf Restore times}
\tabularnewline  \hline
8139too  &  net   & PCI   & 33$\mu$s & 62$\mu$s	\tabularnewline \hline
e1000    &  net   & PCI   & 32$\mu$s & 280ms 	\tabularnewline \hline
r8169    &  net   & PCI   & 26$\mu$s & 30$\mu$s  	\tabularnewline \hline
pegasus  &  net   & USB   & 0$\mu$s & 4ms 	\tabularnewline \hline
ens1371  &  sound & PCI   & 33$\mu$s & 111ms 	\tabularnewline \hline
psmouse  &  input & serio & 0$\mu$s &  390ms 	\tabularnewline \hline

\end{tabular}
\caption{\small \bf Latency for device state checkpoint/restore.}
\label{table:chkrestoretimes}
\vspace{-0.2in}
\end{table}

We examine the device checkpoint latencies to understand the source of
our recovery performance in the previous section.
Table~\ref{table:chkrestoretimes} shows the latency of a checkpoint or
restore for the same six drivers. Checkpointing is very fast, taking
only 20$\mu$s on average and 33$\mu$s at worst. Thus, it is fast
enough to be called frequently, such as before the invocation of most
driver entry points. Restore times are longer, with a range from
30$\mu$s for the r8169 network driver to 390ms for psmouse. USB
drivers store less state because the USB bus controller stores
configuration information instead of the driver. Thus, during a normal
resume, the bus restores configuration state before calling the driver
to resume.  The psmouse driver represents a legacy device and does not
support suspend/resume. Instead, we re-use existing driver code to
reset the mouse.
% measured using the high precision TSC processor register
%%%
\subsection{Utility of Fine Granularity}
We evaluate whether selectively isolating specific entry points is
useful by looking for evidence that driver bugs are confined to one or
a few entry points.  If the functions with bugs are reachable through
a large number of entry points, then full driver isolation is more
useful than per-entry point isolation. For example, if a bug occurs in
a low level read routine, then the bug will affect a large number of
entry points.

In order to have a large data set, we use a published list of hardware
dependence bugs~\cite{kadav09:carb} that represent one of the larger number of unfixed
bugs in the drivers~\cite{hwbugs:lwnarticle}.  We were able to map
these bugs in 210 drivers (541 total bugs) to our kernel under
analysis.  For each driver, we calculate the number of entry points
and the fraction of code in the driver that must be isolated.

We find that the bugs are reachable through 643 entry points, for an
average of 3 per driver. As a comparison, these drivers have a total
of 4,460 entry points (21 per driver), so only 14\% of entry points
must be isolated. The code reachable from these entry points comprises
only 18\% of the code in these drivers. These results indicate that at
least some classes of driver bugs are confined to a single entry
point, and therefore FGFT can reduce the cost of
fault tolerance as compared to isolating the entire driver.
%%%

\subsection{Device Checkpoint Support}
\label{sec:eval:checkpoint}

\newcolumntype{R}{>{\raggedleft} m{0.5in}} % right justified
\newcolumntype{S}{>{\raggedleft} m{0.3in}} % right justified
\newcolumntype{C}{>{\centering} m{0.5in}}
\newcolumntype{L}{>{\raggedright} m{0.4in}} % left justified

\begin{table}[tbp]
\center
\footnotesize

\begin{tabular}{|L|L|R|SS|} \hline

\centering {\bf Class} &
\centering {\bf Bus} &
\centering {\bf Drivers reviewed} &
\multicolumn{2}{c|}{\centering \bf Drivers with PM}
\tabularnewline \hline \hline
net    & PCI     & 104 & 68 & 65\%  \tabularnewline \hline
net    & USB     & 32  & 27 & 84\%  \tabularnewline \hline
net    & PCMCIA  & 4   & 4  & 100\%  \tabularnewline \hline
\hline
sound  & PCI     & 72  & 63 & 88\%  \tabularnewline \hline
sound  & USB     & 3   & 1  & 33\%  \tabularnewline \hline
sound  & PCMCIA  & 2   & 2  & 100\% \tabularnewline \hline
\hline
ATA    & PCI     & 61  & 45 & 74\%  \tabularnewline \hline
SCSI   & USB     & 1   & 1  & 100\% \tabularnewline \hline
SCSI   & PCMCIA  & 5   & 5  & 100\% \tabularnewline \hline 
\hline
{\bf Total} & {\bf - }   &  {\bf  284} &  {\bf  216} & {\bf \raggedleft 76\%}  \tabularnewline \hline

\end{tabular}
\caption{\small \bf Drivers with and without power management as measured 
with static analysis tools. USB drivers (audio and storage) support hundreds of devices
with a common driver, and support suspend/resume.}
\label{table:pmsupport}
\vspace{-0.2in}
\end{table}

Device state checkpointing relies on existing power-management
code. We measure how broadly it applies to Linux drivers by counting
the number of drivers with power-management support. While modern
ACPI-compliant systems require that all devices support power
management, many legacy drivers do not.

We perform a simple static analysis over all network, sound, and storage drivers using the
PCI, USB, and PCMCIA bus in Linux 2.6.37.6. The analysis scans driver entry points and
identifies power management callbacks.  Table ~\ref{table:pmsupport} shows the number of
drivers scanned by class and bus and the number that support power management. Overall, we
found that 76\% of the drivers support power management. Of the drivers that do not
support power management, most were either very old, from before Linux supported power
management, or worked with very simple devices. Only two modern devices, both Intel 10gb
Ethernet cards, did not provide suspend/resume. Thus, we find that nearly all modern
devices support power management and can therefore support device state checkpointing.
%%%
\subsection{Developer effort}

\newcolumntype{R}{>{\raggedleft} m{0.6in}} % right justified
\newcolumntype{C}{>{\centering} m{0.6in}}
\newcolumntype{L}{>{\raggedright} m{0.6in}} % left justified

\begin{table}[tbp]
\center
\footnotesize

\begin{tabular}{|L||R|R|R|} \hline

\multicolumn{1}{|c||}{\multirow{3}{0.6in}{\centering \bf Driver}} &
\multicolumn{1}{c|}{\multirow{3}{0.6in}{\centering \bf Driver LoC}} &
\multicolumn{2}{c|}{\multirow{1}{*}{\centering \bf Isolation annotations}}
\tabularnewline

\cline{3-4}
&
&
\multicolumn{1}{c|}{\multirow{2}{0.6in}{\centering \bf Driver Annotations}} &
\multicolumn{1}{c|}{\multirow{2}{0.6in}{\centering \bf Kernel Annotations}}
\tabularnewline
&
&
&
\tabularnewline \hline

8139too  &  1,904 & 15 & \multirow{3}{*}{20} \tabularnewline \cline{1-3}
e1000    & 13,973 & 32 &     \tabularnewline \cline{1-3}
r8169    &  2,993 & 10 &     \tabularnewline \hline
pegasus  &  1,541 & 26 & 12  \tabularnewline \hline
ens1371  &  2,110 & 23 & 66  \tabularnewline \hline
psmouse  &  2,448 & 11 & 19  \tabularnewline \hline

\end{tabular}
\caption{\small \bf Annotations required by \toolshort\ isolation
  mechanisms for correct marshaling.  Kernel annotations are common to
  a class, and driver annotations are specific to a single driver.}
\label{table:isoprogoverhead}
\vspace{-0.2in}
\end{table}


The primary development cost in using \toolshort\ is adding
annotations, which describe how to copy data between the kernel and
the SFI module. Table \ref{table:isoprogoverhead} shows the number of
annotations needed to apply \toolshort\ to every function in each of
the tested drivers. We separate annotations into {\em driver
  annotations}, which are made to the driver code, and {\em
  kernel-header annotations}, which are a one-time effort common to
all drivers in the class. These annotations are the incremental cost
of making a driver fault tolerant, and the implementation effort of
Isolator and the kernel code described in Section~\ref{sec:device} are
the up-front cost.

Overall, drivers averaged 20 annotations, with more annotations for
drivers with more complex data structures. Most driver classes
required 20 or fewer kernel-header annotations except for sound drivers, which have
a more complex interface and required 66 annotations.
Thus, the effort to annotate a driver is only modest, as annotations
touch only a small fraction of driver code.  In comparison,
SafeDrive~\cite{zca+06} changed 260 lines of code in the e1000 driver
for isolation and another 270 lines for recovery. Nooks~\cite{swift03}
required 23,000 lines of code to isolate and reload drivers. Thus,
these small annotations to drivers may be much simpler than adding a
large new subsystem to the kernel.

\paragraph{Checkpointing implementation.}

\newcolumntype{R}{>{\raggedleft} m{0.6in}} % right justified
\newcolumntype{S}{>{\raggedleft} m{0.3in}} % right justified
\newcolumntype{C}{>{\centering} m{0.5in}}
\newcolumntype{L}{>{\raggedright} m{0.4in}} % left justified

\begin{table}[tbp]
\center
\footnotesize

\begin{tabular}{|c||R|R|}
\hline

\multicolumn{1}{|c||}{\multirow{2}{*}{\bf Driver}}
&
\multicolumn{2}{c|}{\multirow{1}{*}{\bf Recovery additions}}
\tabularnewline

\cline{2-3}

& {\bf LOC Moved}  & {\bf LOC Added} \tabularnewline \hline
8139too   & 26 & 4      \tabularnewline \hline
e1000     & 32 & 10     \tabularnewline \hline
r8169     & 17 & 5      \tabularnewline \hline
pegasus   & 22 & 4      \tabularnewline \hline
ens1371   & 16 & 6      \tabularnewline \hline
psmouse   & 19 & 6      \tabularnewline \hline

\end{tabular}
\caption{\small \bf Developer effort for checkpoint/restore driver callbacks.}
\label{table:recoprogoverhead}
\vspace{-0.22in}
\end{table}

We evaluated the ease of implementing device state checkpointing by
adding support to the six drivers listed in Table
\ref{table:recoprogoverhead}. For each driver we show the amount of
code we copied from suspend/resume to create checkpoint/restore as
well as the number of new lines added. On average, we moved 22 lines of
code and added six lines. The new code adds support for
checkpoint/restore in interrupt contexts and avoids nested locks
when the routines are invoked with a lock held. Even though device
state checkpointing requires adding new code, the effort is mostly
moving existing code. In comparison, implementing a shadow driver
requires (i) building a model of driver behavior and (ii) writing a
wrapper for every function in the driver/kernel interface to log state
changes.







%+ Performance
%-- Zero overhead during regular operation
%-- Performance overhead of isolated I/O 
%+ Fault Tolerance
%-- Fault injection experiments
%-- Demonstrate isolation works
%-- Demonstrate recovery works
%-- Applicability(Coverage) of solution

%+ Recovery Time
%-- Time cost of isolation
%-- Time cost of anticipation
%-- Time cost of recovery

%+ Developer overheads?
%- Ease of generating recovery functions
%- Numbers of annotations required is OK 
%- Working with existing static analysis tools
%- Code patching tools
%-- stack guard
%-- heapify
%-- SFI
%- Bug detection tools
%-- Carburizer
%- No changes to kernel
