\section{Introduction}\label{section:introduction}
A computer system is defined by the data stored in it. The integrity and long term availability of its data is therefore of prime importance. Unfortunately, the persistent data of a computer system stored on hard disks is susceptible to corruptions. There are various reasons for on disk corruptions and many artifacts of literature discuss such on disk corruptions []. Further, bugs in the software stack such as in device drivers and the filesytems themselves also lead to data corruption []. ZFS[] claims to provide novel and robust reliability mechanisms like using checksums in a self validating tree structure. In this project, we aim to perform a type aware corruption analysis of ZFS and also evaluate the need of further types reliability analyses. In particular, by performing type aware corruption analysis we reduce the large search space of random data corruption possible on disk to a representative subset of interesting faults possible and analyse the file system's policy to deal with them.

 Background on ZFS

 - Overview

 ZFS is an object based filesystem and is very differently organized from most regular file systems. ZFS provides transactional consistency and an always on-disk consistency with the help of copy on write semantics and strong checksums which are stored at a different location than the disk blocks.



 ZFS uses the concept of a common storage pools for different filesystems on a system instead of using the traditional concept of volumes. The pool can be viewed as a giant tree sized structure comprising of data at its leaf nodes and metadata at its interior nodes. The stroage pool allocator which manages this pool is responsible for data integrity amongst its host of other functions. This allocator uses a strong checksum mechanism for detection of corruption. For each node (of the tree structure) in the storage pool, the 64 bit checksum of all its children(block pointers) are stored. This mechanism has several advantages like eliminating separate I/O for fetching checksums, fault isolation by separating data and checksum. It also gives a mechanism to store checksum of checksums because of the inherent tree nature of the storage pool. ZFS uses 64-bit second order Fletcher checksums to all user data and 64-bit fourth order Fletcher checksums to all metadata. Using this checksum mechanism, ZFS is able to detect and correct bit rot, misdirected reads, phantom writes, misdirected writes and some user errors. In addition to the above mechanism, ZFS also provides automatic repairs in mirrored configurations and also provides a disk scrubbing facility to detect the latent sector errors while they are recoverable~\cite{Bonwick:Onepager}. By providing these comprehensive reliability mechanisms, ZFS claims to eliminate the need of fsck. To summarize, ZFS presents a data integrity model comprising of an always valid on-disk state therby giving ``no-window of vulnerability''.


 We look at some of the features and specification regarding reliability and also take a look on the disk specification of ZFS.  This help us in understanding the corruption experiments and results in a better manner. ZFS presents a pooled storage model which is unique. The pool comprises of various filesystems and organizes itself in the form of a tree structure. ZFS, thus also performs the function of an Logical Volume Manager which is significant.
 <picture of pool ?>
 ZFS is transcational based object file system. Every structure in ZFS space is an object. The operations on ZFS objects are grouped and is referred to as an transaction group. Another important feature is that all data and metadata blocks are checksummed. The checksums are stored in the parent blocks of the tree like structure, thereby increasing their reliability. It is significant here that all blocks are checksummed, since checksums are considered traditionally expensive and ZFS is one of the the first commercial file system to have end to end checksums. This also rules out the chances of silent corruptions creeping in. Also worth noting is the fact that all modifications to the ZFS structures are copy on write(COW). This always maintains an on-disk consistent state for ZFS. All the data and metadata blocks are compressed in ZFS. While one can disable the data compression, metadata cannot be stored decompressed. This is understandable for this 128 bit, copy-on-write filesystem which duplicates (and even triplicates) all its metadata for reliability purposes.

 ZFS organization

 Entire ZFS file system is represented in terms of either objects or object sets. The objects are represented on disk as dnode_phys_t structure while an object set is represented asan array i.r. dnode_phys_t[]. These structures contain a bonus buffer described by dn_bonus which describes specific information about each object. This is analogus to a programming language where each object(dnode_phys_t) is a template, where different types are represented by specific attributes(dn_bonus). These objects are stored on disk in terms of blocks and these objects connect to one another using block pointers. Block pointers are not traditional pointers in ZFS but are represented by a specific structure called as blkptr_t. This pointer gives us a fair idea about what to expect about the block (or sequence of blocks) being read. The pointer tells us the checksum and compression algorithm information of the next block being read. It also tells us the sizes before and after compression of the logical blocks and whether the blocks are gang blocks(..... of blocks). The addresses of the location of the blocks on disk are calculated using a combination of fields in the block pointer. This address is called as the Data Virtual Address(DVA). The block pointer can store upto three copies of the data pointed by a unique dva. These blocks are referred to as ``ditto'' blocks in ZFS terminology. The ability of block pointers to store multiple valid dvas is referred to as the wideness of the block pointer. This wideness is three for pool wide metadata, two for file system wide metadata and one for data(configurable).
 <picture of blkptr_t -- maybe more important for a pointer corruption paper.>

 Need to check the importance of ZAP.
 Another important structure worth describing is ZAP (ZFS attribute processor). The objects are used to store various attributes to handle arbitary (name, object) associations within an object set (object set). Common attributes include contents of a directory or specific pool wide metadata information.

 To organize the above information in terms of a filesystem or pool, we summarize as follows. Everything in ZFS is an object. The objects when represented as an array form object sets. Filesystems, volumes, clones and snapshots are each examples of objectsets. A snapshot is point in time copy of the filesystem while a clone is a child copy of a snapshot. To delete a snapshot, we need to first delete all its associated clones. A dataset is a collection of multiple objectsets along with spacemaps and snapshot information. A dataset directory groups datasets and provides properties such as quotas, compression etc. Various dataset relationships are also encapsulated using a dataset directory.


 ZFS Figure

 The figure 4 represents the ZFS ondisk structure with one filesystem `myfs'. The figure illustrates the traversal starting from the vdev (virtual device) labels to the data blocks. The figure is divided in two portions using dotted lines. The top half represents the pool wide metadata while the lower half represents the file system metadata and data. As discussed above the pool wide metadata has three copies of the same block. All metadata is compulsarily compressed in ZFS. Only data compression can be disabled here. Also, all blocks are checksummed in ZFS and stored with the parent. As explained above, the block pointers contain checksum information which is compared with the block being read. Upon checksum mismatch, ZFS reads the shadow copy of the corrupted block.

 ZFS On-disk Structures


 The storage pool (ZFS formatted disk) contains four copies of vdev labels. These labels are identical in contents and can be interchangabily used to verify pool contents. vdev labels are the only contents in the pool that have fixed location ondisk (two on front of disk and two on back) and do not have copy on write semantics. ZFS does two staged transactional updates to ensure on-disk consistency. To overwrite any label data, the even labels are updated first followed by update of the odd labels. To access pool information, we need to locate the latest uberblock. Semantically, uberblock is like linux superblock, with a catch that  uberblock contains information about the entire pool and not just one file system. Each vdev contains an array of 128 uberblocks each of size 1K each. ZFS locates the latest uberblock using the transaction group number. The uberblocks are located in vdev in the uberblock array at transaction group number mod 128 th location. The uberblock contains pointer to the root block pointer also known as the metaobject set (MOS). This represents the top of a pool wide mechanism for describing and managing relationships between and properties of various object sets. This is referred to as DataSet and Snapshot layer.  There is only one MOS per pool pointed to by the root block pointer. The MOS also contains the pointer to ZIL or the ZFS Intent Log which contains the transaction records for ZFS's intent logging mechanism.

 As we traverse towards the filesystem metadata, we move from MOS to object directory. The object directory is always fixed and pointed to by the second element of the dnode array. The object directory is a ZAP object and contains references to various other objects which can be located by traversing through the object references in this ZAP. Common references as indicated by the figure include - a)root_dataset - This is a pointer to the root dataset directory. This dataset directory (as described in Section x.x) contains references to all top level datasets in the pool. b)config -This is a name/value list describing the pool vdev configuration. c)sync_bplist - This object contains the list of block pointers that need to be freed during the next transaction. The root dataset contains a reference to additional child directory. The child directory contains the list of file system datasets including the one for our filesystem i.e. "myfs". We also found that microzap always contains a default dataset called as $MOS. We traverse the myfs object from its reference in the objectset and get a pointer to the myfs active dataset. From this active dataset, we get a blockpointer into the filesystem and we now move towards file system specific metadata. All the above references are stored as objects(dnode_phys_t) in the object set array which has three copies in the system.

 As we move towards the file system metadata(object set) from the filesystem's block pointer, we encountered a tree of seven indirect blocks. As our pool and more importantly filesystem just consisted of few files we found it to be pointing to just one objectset, as pointed to by the left most leaf of the chain of indirect pointers. There are two redundant copies of this filesystem object set. We not start traversing this objectset in a similar fashion as we traversed the pool object set.The second entry in this object set contains the master node object. This master node object contains various properties in its ZAP(much similar to how the second entry in pool's object set contained properties like root_dataset, config etc.). The ZAP here gives us a reference(in filesystem (myfs) objectset) to the root directory. The root directory reference contains a object. As we discussed above, the contents of directory are stored in a ZAP. A traversal of this ZAP obtained from this root directory object and their object references in the objectset array. The file object contains the block pointer to the data block. We read this datablock and thus the file contents.

 To summarize, ZFS offers layers of object based indirection which gives sufficient opportunity to perform corruption on this file system. ZFS offers significant reliability features like end to end checksum, concept of wideness and ditto blocks. There metadata corruption also restricts our type aware corruption scope. ZFS is also copy on write, which helps it always maintain an on-disk consistent state. There is also an additional scrub facility provided, which is user initiated and traverses all blocks in the file system and corrects the corrupted blocks based on checksum. In our next section we describe our methodology to test this strong reliability framework followed by results and conclusion. We also describe additional future work based on additional areas of reliability testing based on the knowledge gained from this project.


