Project 4: A Microkernel-based File System (MicroFS)

Project Due Date: Thursday, May 9 at 11:59 PM

0 Updates and Notes

5/4: For the extra credit, you only should support hard links to files (and not to directories).

4/29: File_Read(), File_Write(), and Dir_Read() have all had their argument of type 'pointer to character' changed to a 'pointer to void', which is more appropriate as the arguments are not strings.

4/27: If File_Write() only partially succeeds (i.e. some of the file got written out, but then the disk ran out of space), it is OK to return -1 and set osErrno appropriately.

4/25: File_Create() has been changed: if the file already exists, just return an error (instead of truncating the file). See below for details.

4/25: You should not allow a directory and file name conflict in the same directory (i.e. a file and a directory of the same name in the same directory).

4/25: Dir_Read() has been changed to read in directory entries (both name and inode number), and not just names. See below for details.

4/25: Assuming that the maximum name length of a file is 16 bytes means 15 characters plus one for an end-of-string delimiter.

4/25: LibOS.h and API have changed slightly to update corrections (i.e. Dir_Unlink() takes a pathname).

4/23: The maximum length of a path is 256 characters.

4/23: The maximum number of open files is 256.

4/23: Legal characters for a file name include letters (case sensitive), numbers, dots ("."), dashes ("-"), and underscores ("_").

4/23: For the extra credit about hard links, you will need to implement the interface Link_Create(). The prototype for this routine is now available in LibOS.h. Upon a failure, this routine should return -1 and set osErrno to E_CREATE, and upon success, it should return 0.

4/23: You should enforce a maximum limit of 1000 inodes. Define a constant internal to your OS called MAX_INODES and make sure you observe this limit when allocating new inodes for either files or directories.

1 Objective

When you are finished with this project, you should have a better understanding of how a file system works, including directory hierarchy management and storage management issues.

2 Overview

This project will again include our two old friends, the user-level OS, and the library interface to OS services, LibOS. This time, we will be building a real honest-to-goodness file system within the user-level OS, including directories and normal files.

Of all of the projects so far, this one is by far the most involved, so make sure to start early! A lot of coding is involved, as well as careful design work to make sure your overall structuring will be satisfactory.

One other thing: your OS, while not needing to support most previous functionality, should still support FIFO scheduling. This should simplify your system (in that you don't have to worry about concurrent processes accessing the file system).

3 LibOS API

We'll start by describing the LibOS API to the file system. There are four parts to the API: the standard OS_Init() and OS_Fini() routine, a set of calls that deal with file access, a set of calls that deal with directories, and one generic file system call.

3 . 1 Basic OS API

int OS_Init(char *file)
OS_Init() contacts the OS bound at file and informs the OS of the calling process's presence in the system. OS_Init() should not return until the process is running, as per FIFO scheduling. Thus, OS_Init() should wait for a return message from the OS before it returns to the user. Upon success, return 0. Upon failure, set osErrno to E_GENERAL and return -1.

int OS_Fini()
OS_Fini() informs the OS that the process is about to exit. The OS can then take any needed action to clean up the process state and perhaps begin running another process. Unlike before, OS_Fini() should just exit upon successful contact with the OS. Thus, OS_Fini() never returns successfully. Upon failure, set osErrno to E_GENERAL and return -1.

3 . 2 File Access API

int File_Create(char *file)
File_Create() creates a new file of the name pointed to by file. If the file already exists, you should return -1 and set osErrno to E_CREATE. Note: the file should not be "open" after the create call. Rather, File_Create() should simply create a new file on disk of size 0. Upon success, return 0. Upon a failure, return -1 and return E_CREATE.

int File_Open(char *file)
File_Open() opens up a file (whose name is pointed to by file ) and returns an integer file descriptor (a number greater than or equal to 0), which can be used to read or write data to that file. If the file doesn't exist, return -1 and set osErrno should be set to E_NO_SUCH_FILE. If there are already a maximum number of files open, return -1 and set osErrno to E_TOO_MANY_OPEN_FILES.

int File_Read(int fd, void *buffer, int size)
File_Read() should read size bytes from the file referenced by the file descriptor fd. The data should be read into the buffer pointed to by buffer. All reads should begin at the current location of the file pointer, and file pointer should be updated after the read to the new location. If the file is not open, return -1, and set osErrno to E_BAD_FD. If the file is open, the number of bytes actually read should be returned, which can be less than or equal to size. (The number could be less than the requested bytes becuase the end of the file could be reached.) If the file pointer is already at the end of the file, zero should be returned, even under repeated calls to File_Read().

int File_Write(int fd, void *buffer, int size)
File_Write() should write size bytes from buffer and write them into the file referenced by fd. All writes should begin at the current location of the file pointer and the file pointer should be updated after the write to its current location plus size. Note that writes are the only way to extend the size of a file. If the file is not open, return -1 and set osErrno to E_BAD_FD. Upon success of the write, all of the data should be written out to disk and the value of size should be returned. If the write cannot complete (due to a lack of space), return -1 and set osErrno to E_NO_SPACE. Finally, if the file exceeds the maximum file size, you should return -1 and set osErrno to E_FILE_TOO_BIG.

int File_Seek(int fd, int offset)
File_Seek() should update the current location of the file pointer. The location is given as an offset from the beginning of the file. If offset is larger than the size of the file or negative, return -1 and set osErrno to E_SEEK_OUT_OF_BOUNDS. If the file is not currently open, return -1 and set osErrno to E_BAD_FD. Upon success, return the new location of the file pointer.

int File_Close(int fd)
File_Close() closes the file referred to by file descriptor fd. If the file is not currently open, return -1 and set osErrno to E_BAD_FD. Upon success, return 0.

int File_Unlink(char *file)
This should delete the file referenced by file, including removing its name from the directory it is in, and freeing up any data blocks and inodes that the file was using. If the file does not currently exist, return -1 and set osErrno to E_NO_SUCH_FILE. If the file is currently open, return -1 and set osErrno to E_FILE_IN_USE (and do NOT delete the file). Upon success, return 0.

3 . 3 Directory API

We now describe the directory operations you must implement.

int Dir_Create(char *path)
Dir_Create() creates a new directory as named by path. In this assignment, all paths are absolute paths, i.e., you can assume that you don't have to track the current working directory or anything like that. Creating a new directory takes a number of steps: first, you have to allocate a new file (of type directory), and then you have to add a new directory entry in the current directory's parent. Upon failure of any sort, return -1 and set osErrno to E_CREATE. Upon success, return 0.

int Dir_Size(char *path)
Dir_Size() returns the number of bytes in the directory refered to by path. This routine should be used to find the size of the directory before calling Dir_Read() (described below) to find the contents of the directory.

int Dir_Read(char *path, void *buffer, int size)
Dir_Read() can be used to read the contents of a directory. It should return in the buffer a set of directory entries. Each entry is of size 20 bytes, and contains 16-byte names of the directories and files within the directory named by path, followed by the 4-byte integer inode number. If size is not big enough to contain all of the entries, return -1 and set osErrno to E_BUFFER_TOO_SMALL. Otherwise, read the data into the buffer, and return the number of directory entries that are in the directory (e.g., 2 if there are two entries in the directory).

int Dir_Unlink(char *path)
Dir_Unlink() removes a directory referred to by path, freeing up its inode and data blocks, and removing its entry from the parent directory. Upon success, return 0. Note: Dir_Unlink() should only be successful if there are no files within the directory. If there are still files within the directory, return -1 and set osErrno to E_DIR_NOT_EMPTY. If someone tries to remove the root directory ("/"), don't allow them to do it! Return -1 and set osErrno to E_ROOT_DIR.

3 . 4 Generic File System API

int FS_Sync()
FS_Sync() makes sure the contents of the file system are stored persistently on disk. More details on how this is accomplised are available below. Upon success, return 0. Upon failure, return -1 and set osErrno to E_GENERAL.

3 . 5 Some Notes

When reading or writing a file, you will have to implement a notion of a current file pointer. The idea here is simple: after opening a file, the current file pointer is set to the beginning of the file (byte 0). If the user then reads N bytes from the file, the current file pointer should be updated to N. Another read of M bytes will return the bytes starting at offset N in the file, and up to bytes N+M. Thus, by repeatedly calling read (or write), a program can read (or write) the entire file. Of course, File_Seek() exists to explicitly change the location of the file pointer.

Note that you do not need to worry about implementing any functionality that has to do with relative pathnames. In other words, all pathnames will be absolute paths. Thus, all pathnames given to any of your file and directory APIs will be full ones starting at the root of the file system, i.e., /a/b/c/foo.c. Thus, your file system does not need to track any notion of a "current working directory".

4 Implementing the File System

4 . 1 The Disk Abstraction

One of the first questions you might ask is "Where am I going to store all of the file system data?" A real file system would store it all on disk, but since we are writing a user-level operating system, we will store it all in a "fake" disk, provided to you at no cost. In Disk.c and Disk.h you will find the "disk" that you need to interact with for this assignment.

The "disk" that we provide presents you with NUM_SECTORS sectors, each of size SECTOR_SIZE. Thus, you will need to use these values in your file system structures. The model of the disk is quite simple: in general, the file system will perform disk reads and disk writes to read or write a sector of the disk. In actuality, the disk reads and writes access an in-memory array for the data; other aspects of the disk API allow you to save the contents of your file system to a regular Solaris file, and later, restore the file system from that file.

Here is the basic disk API:

int Disk_Init()
Disk_Init() should be called exactly once by your OS before any other disk operations take place.

int Disk_Load(char* file)
Disk_Load() is called to load the contents of a file system in file into memory. This routine will probably be executed once by the OS when it is "booting".

int Disk_Save(char* file)
Disk_Save() saves the current in-memory view of the disk to a file named file. This routine may end up getting executed somewhat frequently, to assure that changes to the disk do not get lost if the OS crashes.

int Disk_Write(int sector, char* buffer)
Disk_Write() writes the data in buffer to the sector specified by sector. The buffer is assumed to be of size sector exactly.

int Disk_Read(int sector, char* buffer)
Disk_Read() reads a sector from sector into the buffer specified by buffer. As with Disk_Write(), the buffer is assumed to be of size sector exactly.

For all of the disk API: All of these operations return 0 upon success, and -1 upon failure. If there is a failure, diskErrno is set to some appropriate value -- check the code in Disk.c for details.

4 . 2 On-Disk Data Structures

A big part of understanding a file system is understanding its data structures. Of course, there are many possibilities, but for this assignment, we recommend the following.

First, somewhere on disk you need to record some generic information about the file system, in a block called the superblock. This should be in a well-known position on disk -- in this case, make it the very first block. For this assignment, you don't need to record much there. In fact, you should record exactly one thing in the superblock -- a magic number. Pick any number you like, and when you initialize a new file system (as described in the booting up section below), write the magic number into the super block. Then, when you boot up with this same file system again, make sure that when you read that superblock, the magic number is there. If it's not there, assume this is a corrupted file system (and that you can't use it).

To track directories and files, you are going to need two types of blocks: inode blocks and data blocks. First, let's examine inodes. In each inode, you need to track at least two things about each file. First, you should track the size of the file. Second, you need to track the type of the file (normal or directory). Third, you should track which file blocks got allocated to the file. For this assignment, you can assume that the maximum file size is 30 blocks. Thus, each inode should contain 1 integer (size), 1 integer (type), and 30 pointers (to data blocks). You might also notice that each inode is likely to be smaller than the size of a disk sector -- thus, you should put multiple inodes within each disk sector to save space.

Second, there are data blocks. Assume that each data block is the exact same size as a disk sector. Thus, part of disk must be dedicated to these blocks.

Of course, you also have to track which inodes have been allocated, and which data blocks have been allocated. To do this, you should probably use a bit map for each, i.e., the first block after the superblock should be the inode bitmap, and the second block after the superblock should be the data block bitmap.

To see a full picture of what the disk layout should look like, check out this picture.

4 . 3 Path Lookup

One of the hard parts about this file system will be pathname lookup. Specifically, when you wish to open a file named /foo/bar/file.c, first you have to look in the root directory ("/"), and see if there is a directory in there called "foo". To do this, you start with the root inode number (which should be a well-known number, like 0), and read the root inode in. This will tell you how to find the data for the root directory, which you should then read in, and look for foo in. If foo is in the root directory, you need to find it's inode number (which should also be recorded in the directory entry). From the inode number, you should be able to figure out exactly which block to read from the inode portion of the disk to read foo's inode. Once you have read the data within foo, you will have to check to see if a directory "bar" is in there, and repeat the process. Finally, you will get to "file.c", whose inode you can read in, and from there you will get ready to do reads and writes.

4 . 4 Open File Table

When a process opens a file, first you will perform a path lookup. At the end of the lookup, though, you will need to keep some information around in order to be able to read and write the file efficiently (without repeatedly doing path lookups). This information should be kept in a per-file open file table. When a process opens a file, you should allocate it the first open entry in this table -- thus, the first open file should get the first open slot, and return a file descriptor of 0. The second opened file (if the first is still open) should return a descriptor of 1, and so forth. Each entry of the table should track what you need to know about the file to efficiently read or write to it -- think about what this means and design your table accordingly. It is OK to limit the size of your table to a fixed size.

4 . 5 Disk Persistence

The disk abstraction provided to you above keeps data in memory until Disk_Save() is called. Thus, you will need to call Disk_Save() to make the file system image persistent. A real OS commits data to disk quite frequently, in order to guarantee that data is not lost. However, in this assignment, you only need to do this when FS_Sync() is called.

4 . 6 Booting Up

When "booting" your OS (i.e., starting it up), you will be passed a filename that is the name of your "disk". That is, it is the name of the file that holds the contents of your simulated disk. If the file exists, you will want to load it (via Disk_Load()), and then check and make sure that it is a valid disk. For example, the file size should be equivalent to NUM_SECTORS times SECTOR_SIZE, and the superblock should have the information you expect to be in there (as described above). If any of those pieces of information are incorrect, you should report an error and exit.

However, there is one other situation: if the disk file does not exist, this means you should create a new disk and initialize its superblock, and create an empty root directory in the file system. Thus, in this case, you should use Disk_Init() followed by a few Disk_Write() operations to initialize the disk, and then a Disk_Save() to commit those changes to disk.

4 . 7 Other Notes

Directories: Treat a directory as a "special" type of file that happens to contain directory information. Thus, you will have to have a bit in your inode that tells you whether the file is a normal file or a directory. Keep your directory format simple: a fixed 16-byte field for the name, and a 4-byte entry as the inode number.

Maximum file size: 30 sectors. If a program tries to grow a file (or a directory) beyond this size, it should fail. This can be used to keep your inode quite simple: keep 30 disk pointers in each inode. You don't have to worry about indirect pointers or anything like that (that a real file system would have to deal with).

Maximum element length in pathname: 16 characters. You don't have to worry about supporting long file names or anything fancy like that. Thus, keep it simple and reserve 16 bytes for each name entry in a directory.

Maximum number of inodes: You should enforce a maximum limit of 1000 inodes. Define a constant internal to your OS called MAX_INODES and make sure you observe this limit when allocating new inodes for either files or directories.

Maximum length of a path: 256 characters.

Maximum number of open files: 256.

Legal characters for a file name: letters (case sensitive), numbers, dots ("."), dashes ("-"), and underscores ("_").

5 Extra Credit

Hard links. Implement hard links as described in class and in the text. This will require an extra field in the inode for a reference count (you can probably split the integer reserved for the "type" field into two shorts), and some extra work to figure out when it is actually OK to free the inode and data. The interface to create the hard link is called Link_Create(), and is found in LibOS.h. Upon a failure, this routine should return -1 and set osErrno to E_CREATE, and upon success, it should return 0. You should only support links to files, and not to directories.

Name cache. Path lookups are expensive. Instead, implement a name lookup cache that takes a pathname and returns the inode number of the file. The cache should be a fixed-size, storing 10 pathnames, and should be managed in LRU fashion.

6 The OS Interface

You should make the OS runnable as follows:



prompt> ./os -f filename -d diskname

The -f filename flag is required, and passes to the OS the name of the file that it will be bind to; other processes will use this name to direct their messages to the OS process. The -d diskname tells the OS which file to use as the "disk". If the "disk" does not exist, you should create a new disk of the proper size, and initialize it for use, as described above.

Again, feel free to use getopt.c to make your command-line parsing a little cleaner and simpler.

7 Program Design and Implementation

Once again, before writing a single line of code, both partners should sit down together and design the entire system. This is so important that it will be repeated, this time in bold. Before writing a single line of code, both partners should sit down together and design the entire system.

Now that we have that out of the way, here are some suggestions on how to approach this.

Read the entire project description at least

twice,

Meet with partner at least once to come up with some preliminary designs. At this point, both partners should be looking at how all the major components should work and fit together. This should be from a very high level. Don't worry about the tiny details here. Some people feel that doing a flow chart at this stage (showing how the different components are to interact) is very helpful in doing the next step.
Decide on which partner should do the detailed designs for which parts. This will be more difficult for this project, so spend some time thinking about it.
Completed detailed design. Both partners should have a written description of all the functions they see being needed. This description should also include psuedo-code describing what each function should do. Remember, no code has been written at this point - you are designing the system first! When both people have their halves done, the partners should sit down and review each other's work, make suggestions, corrections, modifications, etc. Of course, if you've been in touch with each other throughout, this part goes much more smoothly - there are few suprises in store for your partner.
Start coding. First, write small pieces of code that get you familiar with the new functionality. Then, slowly move towards the desired end-goal, testing as you go.

DO NOT WRITE ALL OF THE CODE WITHOUT TESTING AS YOU GO.

Put it all together. If you've really taken the time before this to design a good system, you will be amazed at how easily this step goes. Of course you will run into a few snags but they should be few and quickly dispatched. Believe it or not, many professional software developers will tell you that coding is the easiest stage in any software development. The only reason for that, however, is because they do their homework up front designing the system.
Hand it in. Let out a barbaric yawp -- you are finished with 537 projects forever! (probably)

There are three major points about the above stategy. Number one, stay in touch with your partner. Do not divide the work and then speak to each other only the day before the due date. Stay in touch! Number two, work hard to develop a good design before writing any code. It is very hard to over-emphasize this point (although I'm trying hard). And lastly, get started early. Don't wait until the last week. We give you three weeks to do these projects for a reason - they take that long.

8 Provided Materials (Summary)

The following files have been provided here for you.

The source file for the disk abstraction:

Disk.c

The header file for the disk abstraction:

Disk.h

Domain communication source code:

Domain.c

Domain communication header file:

Domain.h

The LibOS header file (Write your own LibOS.c):

LibOS.h

Processing arguments with getopt:

getopt.c

9 Handing in Your Project

The directory for handing in your program can be found at:

~cs537-1/handin/(username)/p4

where (username) is your login. You only need to put copies of your code into one partner's handin dirctory.

You should only hand in the files that you created and/or modified. You should probably also include Domain.c, Domain.h, and all of that other stuff that is required so we can just type make and build the entire darn thing. You should also submit the Makefile needed to build your program. Lastly, don't forget to hand in a README file that indicates how to run your program, known bugs, the names of both partners, and any other information you that is important to runnning your program.

10 Grading

No late assignments will be accepted. This project is due on Thursday, May 9 at 11:59 PM.

This assignment will be graded based on correctness of implementation as well as robustness. This means your program should work under all the test cases all the time. Programs that only partially work or fail intermittently will be penalized.

If you do not have a fully functional program, it is your responsibility to be able to quickly and efficiently show which of the above functionality is working properly. For example, to show that you are creating and terminating processes correctly at the OS, you could print out the entire runnable queue every time a new process enters or leaves the system.