CS 537 Spring 2007
Programming Assignment 5
File Systems, Part II
Directories

Due:

Friday, May 11, 2003 at 1:00 am.
There will be no extensions on the due date for this project

Last updated: Mon May 7 10:05:50 CDT 2007.

May 7: I have updated FileTester.java to make it more robust in the case of an empty argument to the write command or a very long argument to write or writeln. I also fixed a bug in pwd in the case that the free map requires more than one block. Thanks to Joshua Royalty for finding this bug and suggesting a fix.
The new version is available in ~cs537-1/public/project5/FileTester.java.
May 1: There was a bug in FileTester.java that caused it to die on the read command when the file contained non-printing characters. Line 314 was
```
    out.printf("\\%03o", Integer.toString(b, 8));
```
It should be
```
    out.printf("\\%03o", b);
```
A corrected version is available in ~cs537-1/public/project5/FileTester.java.
There is a FAQ (Frequently Asked Questions) page associated with this project. Please check it frequently for updates.
April 25: The supporting source files are now available. To get started,
```
    cd ~/private
    mkdir p5
    cd p5
    cp ~cs537-1/public/project5/* .
    make
```
The following documentation has been revised as follows:
- The FileTester accepts a pwd command.
- The argument to ls is optional (it defaults to “.”).

Introduction
Getting Started
File System API
Implementation
The Command Interpreter
Implementation Hints
Grading
What to hand in

Introduction

In the previous assignment, you built a buffer cache and disk scheduler for the MiniKernel. In a real OS, you know that users cannot modify blocks willy-nilly -- they must request access to data through the file system. In this assignment, you will start with a fresh Kernel and build a simple file system.

Getting Started

First, download a fresh copy of the MiniKernel sources. This version has a new, improved version of Disk called FastDisk.java. It also has eleven new methods added to Library.java. As written, they all return error indications to indicate that they are not yet implemented. It is your job to implement them.

FastDisk.java differs from Disk.java in that it has a buffer cache and elevator scheduling algorithm built in. (Actually, that's a lie. It just doesn't simulate any delays, but you can pretend that it has buffering and scheduling built in.) The methods beginRead and beginWrite of Disk.java have been replaced by new methods read and write that perform the operation so quickly that when they return, the operation is complete. There is no need to do any buffering or scheduling of your own, and no need to deal with disk interrupts.

Although the new disk is fast, it is not very big. All block numbers are represented as short integers, so the largest possible disk has 32,767 blocks (16 megabytes, if BLOCK_SIZE is 512).

File System API

This filesystem will allow users to read, write, create, and delete files on disk. Because the disk is so small (and to make your life easier), each file, symlink, or directory is just one block long! There is a tree-structured directory system and symbolic links, just like in Unix, but unlike Unix there are no “hard” links. Files are identified by pathnames, which are readable character strings possibly containing slashes (/). If p is a pathname, we use the notation prefix(p) to mean the part of p up to the last slash in p (or “.” if p does not contain any slashes). There is a current working directory (cwd). Unlike Unix, which has a cwd for each process, there is just one cwd that applies to all system calls. Pathnames that start with a slash are absolute pathnames; they are resolved relative to the root directory. Pathnames that do not start with a slash are resolved relative to the current working directory. On startup, the cwd is the root directory.

You must implement the 11 system calls format, chdir, create, read, write, delete, mkdir, rmdir, symlink, readlink, and readdir, described below.

Each of these system calls returns 0 for success and -1 if there is an error. For debugging purposes, you many want to print an error message before returning -1.

int format();
This method initializes the contents of the disk with any data structures necessary to represent an “empty” file system. This method should create an “empty” root directory, containing only the “.” and “..” entries, both of which point to itself.
int chdir(String pathname);
Sets the current working directory to the directory identified by pathname, which may be either absolute or relative. The initial current working directory is the root directory. If pathname does not name a valid existing directory, the cwd is unchanged and an error indication is returned.
int create(String pathname);
Creates a new file with the indicated pathname. The initial contents of the file are all null (zero) bytes. It is an error if prefix(pathname) does not identify an existing directory, the whole pathname identifies an existing file, directory or symlink, or if the disk or directory is full.
int read(String pathname, byte[] buffer);
Reads the contents of the file indicated by pathname into buffer. It is an error if pathname does not name an existing ordinary file, buffer is null, or if buffer.length < Disk.BLOCK_SIZE.
int write(String pathname, byte[] buffer);
Overwrites the file indicated by pathname with data from buffer. It is an error if pathname does not name an existing ordinary file, buffer is null, or if buffer.length < Disk.BLOCK_SIZE.
int delete(String pathname);
Destroys the indicated file or symlink. It is an error if the indicated file does not exist or is a directory. Note that if pathname denotes a symlink, it is the symlink itself, not the file to which it refers, that is deleted.
int mkdir(String pathname);
Creates a new directory with the indicated pathname. The directory starts out “empty,” that is, it contains only two entries: . points to the new directory itself and .. points to its parent directory. It is an error if prefix(pathname) does not identify an existing directory, the whole pathname identifies an existing file, directory or symlink, or if the disk or directory is full.
int rmdir(String pathname);
Deletes the indicated directory. It is an error if pathname does not indicate an existing directory, if the directory is non-empty (contains entries other than . and ..), or if the directory is the root directory.
int symlink(String oldName, String newName);
Creates a new symbolic link with name newName and contents oldName. It is an error if prefix(newName) does not identify an existing directory, newName identifies an existing file, directory or symlink, or if the disk or directory is full. The oldName need not be valid.
int readlink(String pathname, byte[] buffer);
Reads the contents of the symlink indicated by pathname into buffer. It is an error if pathname does not exist or is not a symlink, buffer is null, or if buffer.length < Disk.BLOCK_SIZE.
int readdir(String pathname, byte[] buffer);
Reads the contents of the directory indicated by pathname into buffer. It is an error if pathname does not name an existing directory, buffer is null, or if buffer.length < Disk.BLOCK_SIZE.

Implementation

IMPORTANT: You do not have to worry about race conditions for this project. You may assume there is just one instance of FileTester running, doing one kernel operation at a time. You had enough problems dealing with race conditions on projects 2, 3, and 4.

The first few block of the disk contain a free map, which is a big array of bits, with one bit for each block on the disk. The bit is 1 if the block is currently allocated and 0 if it is free (available for allocation). The blocks of the free map itself are always marked as “allocated”. Each remaining block is marked “allocated” if and only if contains a directory, symlink, or ordinary file. There are 8 bits in a byte, so the map will need diskSize/8 bytes, or diskSize/(8*BLOCK_SIZE) blocks, where this value needs to be rounded up to the nearest integer value. For example, with BLOCK_SIZE == 512 and diskSize == 10000, 10000/(8*512) = 2.44..., so three blocks will be required:

    byte[] freeMap = new byte[3 * BLOCK_SIZE];

Within each byte, bits are numbered from low to high, so the low-order bit of freeMap[0] describes the state of block 0, the next lowest-order bit describes block 1, and so on. In general, block i of the disk is free (unused) if and only if

    ((freeMap[i/8] >> (i % 8)) & 1) == 0

The rest of the blocks on the disk may be allocated to hold directories, symlinks, or ordinary files. The fact that a block is allocated is indicated by the free map, but the kind of file it holds can only be determined by its position in the directory tree. The allocated blocks are organized as a tree. Internal nodes are directories, and the leaves are symlinks and ordinary files. The root directory is the first block following the free map.

A directory consists of a sequence of 16-byte entries. Each entry contains a two-byte block number, a one-byte type flag and a thirteen-byte name. The block number is calcuated by the formula

    b = ((b0 & 0xff) << 8) + (b1 & 0xff),

where b0 and b1 are the first two bytes in the entry. We say that the entry “points to” block b. If b == 0 the entry is unused. The third byte of the entry indicates the type of block b: ‘O’ means “orinary file”, ‘D’ means “directory”, and ‘L’ means “symlink”. The remaining bytes hold the name of the entry, one character per byte. If the name is less than 13 characters long, it is padded with trailing null bytes (bytes with the value 0, not '0' characters). Each directory contains two entries with the names “.” and “..”, which point to the directory itself and the directory's parent in the directory tree. (As a special case, “..” in the root directory points to the root directory). If these are the only allocated entries in the directory, we say the directory is “empty”. With the exception of the root directory block, each allocated block on disk should be pointed to by exactly one directory entry other than “.” or “..”. In other words, this file system does not support “hard links”.

A symlink contains a pathname, with one character per byte. If the the length of the pathname is less than BLOCK_SIZE, it is padded with null bytes. An “ordinary” file contains “user” data. When it is created, it is filled with null bytes. Otherwise, the kernel should make no assumptions about its contents.

Disk routines only work on arrays of bytes. In order to read and write directories and symlinks, you'll have to convert shorts and Strings to bytes, and vice versa. You should write some functions to perform these conversion, because you will do them many times. Here is some code to get you started.


    /** Store a 16-bit integer into a byte array.
     * @param n the integer to be stored
     * @param buf the byte array into which it should be stored
     * @param offset the index of the first byte to be modified
     */
    static void pack(short n, byte[] buf, int offset) {
        buf[offset] = (byte)(n >> 8);
        buf[offset+1] = (byte)n;
    }

    /** Convert a field in a byte array to an integer.
     * @param buf the byte array containing the data.
     * @param offset the location in the array where the data starts.
     * @return the short integer value.
     */
    static short unpackShort(byte[] buf, int offset) {
        return (short) (((buf[offset] & 0xff) << 8) + (buf[offset+1] & 0xff));
    }

To convert between byte arrays and Strings, use the String constructor String(byte[] bytes, int offset, int length) and the method String.getBytes(). When converting from bytes to Strings, be careful not to include any trailing null bytes used for padding.

The Command Interpreter

The Shell of project 4 is replaced in this project with a program called FileTester, which is a command interpreter specifically designed for testing your file system. This program is meant to be specified as the “shell” to the Kernel by typing


    java Boot cacheSize FastDisk size FileTester

where size is the size of the simulated disk, in blocks, and cacheSize is any integer (it is ignored for this project). For example, you might try


    java Boot 1 FastDisk 100 FileTester

If a (Unix) file named DISK exists in the current directory, it should be the result of any earlier run with the same size parameter. Otherwise, a new DISK will be created. The first block will be filled with null bytes, and the rest will contain random data. In this case, you should be careful that the first command you type is format.

You can also run the program to take its commands from a script, as in


    java Boot 1 FastDisk 100 FileTester test1.script

Input lines starting with “/*” or “//” are ignored (the latter are echoed to the output). Other lines have the format

    command [ args ]

Most of the commands correspond to system calls. Type “help” for a list. By longstanding tradition, some of the commands have slightly different names from the methods they call

command method
rm delete
cd chdir
ln symlink

command	method
rm	delete
cd	chdir
ln	symlink

There are also a few special cases.

The help command prints a list of all the commands.
The quit command terminates the program (as does end-of-file).
write fname data ... converts data ... to an array of bytes (padding with nulls up to the block size) and writes it the the file named fname.
writeln fname is similar, but the data comes from the following lines (including their terminating newlines) up to, but not including, a line consisting of a single '.'. If there is not enough data to fill a block, the remainder of the block is cleared to nulls. Excess data bytes are ignored.
read file displays the contents of the indicated file. Null bytes are not displayed.
ls dirname displays the contents directory dirname. The dirname may be omitted, in which case, it defaults to “.”. Subdirectories are indicated by a trailing '/'; symlinks are indicated by '-<' following the contents of the symlink.
pwd prints the current working directory. I uses the readdir system call and interprets the contents of various directories, so it will fail if your readdir implementation is broken or the contents of directories do not follow the specification in the Implementation section.

Implementation Hints

You will want to break this project into pieces, implementing and testing each piece in turn.

Allocating and freeing disk space. Write private methods to allocate and release disk blocks from the free map. You may find it easiest to cache the entire free map in memory, reading it in as part of doPowerOn and flushing it back to disk in doShutdown.
Manipulating directories. You will needs methods to extract the three fields from a directory entry (converting the block number and type into integers and the name to a String), and the reverse operation of inserting the three values into an entry. You will also need methods to search a directory for a given string, and to allocate an unused entry.
Interpreting pathnames. Use a StringTokenizer or String.split(regex) to break up a pathname into its components, and write a method
```
    private short namei(short start, String[] path)
```
to simplify walking the directory tree. This method takes a starting point in the file-system tree (as the block number of a directory) and walks through the file system following the path indicated by the strings in path. The resulting value is a “pointer” to the end of the path.
Symlinks. Modify namei to handle symlinks correctly and add code to implement symlink and readlink. The latter may be a bit tricky.

Grading

Your grade will be 80% correctness, 10% testing, and 10% style. Don't forget the following:

Implement algorithms as described.
Make your code readable, robust, and reusable.
Test a variety of situations, including error conditions.
Consider your test results -- do they make sense?

What to hand in

Copy into your handin directory a complete set of all the .java files needed to run your program. Also copy in any test scripts you created. Also include a file named README containing a line of the form Partner: login1 login2 and a brief description of your project and testing.

Last modified: Mon May 7 10:05:50 CDT 2007

CS 537 Spring 2007 Programming Assignment 5 File Systems, Part II Directories