\section{Implementation}\label{section:implementation}

 We develop a pseudo device driver called ``corrupter'' driver which is a standard Solaris layered driver exposing the LDI (Layered Driver Interface). This pseudo layered driver creates two minor nodes in the /devices tree. One device node is used for block I/O while the other exposes a raw character device.

 We create a storage pool on the block device minor node exposed by the ``corruptor". The corrupter thus interposes between the ZFS Virtual devices (VDEVs) and the disk driver beneath. The kernel interface of the LDI exposed by the corrupter driver is called in by the VDEVs to perform I/O for the pool. The corrupter in turn calls into the actual disk driver using LDI for performing the I/O.

 We also develop an application called the``analyser" which uses the device nodes exposed by driver. The ``analyser" uses the block device minor node for reading the on-disk ZFS data. The ``analyser" reads the on-disk structures and uses the knowledge of these structures to selectively corrupt different types of ZFS on-disk objects. It presents the on-disk picture to the user who can then instruct the ``analyser" to selectively corrupt objects. For corrupting an on-disk object, the ``analyser" finds the physical disk block and offset to corrupt. It then uses the raw character device node to call into the driver via an ioctl(). The ioctl passes a structure to the driver containing the <physical block, offset, new-value> for corruption and the driver adds this to its table of corruption. For performing the corruption, the driver always perform a check on reads and looks for a match in its table of corruption. In case of a match, the driver modifies the given offset with the new value and sends this corrupted block to the calling process.

 We use an application as the ``consumer'' which calls into ZFS using the POSIX filesystem APIs. For different filesystem operations performed by the ``consumer" such as reads and writes, the corrupter driver injects the faults in the blocks read by ZFS. It does this corruption only once for each block. This behavior is representative of the transient errors on disks. Also, when writes are performed back to the identified blocks, the corrupter driver monitors these writes and doesn't further corrupt them on reads. We then analyze the behavior of ZFS to recover from faults injected by our driver. To analyse the behavior of ZFS, we see the return values, system logs and also trace the the system calls.

Corruption Experiments
ZFS uses compression on all its metadata while data compression for files and directories is an optional feature. So, any corruption in the ZFS metadata can be effectively viewed as a block level corruption on the compressed metadata. In the experiments we performed, we corrupted the compressed ZFS metadata objects and analysed the behaviour of ZFS in response to the corruptions. For file and directory data, we corrupted the uncompressed data blocks on disk. We also do not use ditto blocks for the file and directory data. The following is a list of the different objects on which the corruption experiments were performed:

\begin{figure}[tbp]
\begin{center}
\epsfxsize=2.9in
\epsfbox{figs/machines.eps}
\end{center}
\caption{Types of machine organization}
\label{fig:fig1}
\end{figure}

