CS739 Spring 2008: Distributed Systems Instructor: Andrea C. Arpaci-Dusseau When: TR 11:00 - 12:15 Where: 1289 CS Office: 7375 CS Office Hours: Friday 2-3 and by appointment Mailing list archive News Final project due May 12th. Mini-project due in two weeks: April 8th. Paper useful for mini-project: just the part describing sample sort: "Guy E. Blelloch and Charles E. Leiserson and Bruce M. Maggs and C. Greg Plaxton and Stephen J. Smith and Marco Zagha", A Comparison of Sorting Algorithms for the Connection Machine CM-2, Symposium on Parallel Algorithms and Architectures", 1991 Overview Welcome to the graduate distributed systems class. This semester, we will be reading both the classic papers on distributed systems as well as more recent research papers; we are going to be focusing on the distributed system knowledge that is needed to build scalable services like Google. The work in this course will have two components: reading papers and a research project. For reading papers, you will have three responsibilities: writing a short essay to answer a discussion question for each paper, periodically scribing our lecture discussion and recording it for posterity, and presenting a paper with a small group to the rest of the class. Readings This semester, our reading will focus on the distributed system knowledge that is needed to build scalable services like Google. We will be mostly reading papers from the two top systems conferences: Symposium on Operating Systems Principles (SOSP) and Operating Systems Design and Implementation (OSDI). We will also read a few of the most classic distributed system papers to fill in some necessary background. The complete list of papers is available here These papers assume that you already know the relevant background in the operating systems literature. Therefore, I expect that you have already taken CS736 (Graduate Operating Systems) before taking this course; if you have not, then please see me on the first day of class. During class, you will all be expected to participate in our discussion. We will not be having any exams in this course; therefore, your "reading" grade will be based on your paper write-ups, your class participation, your scribing, and your group presentation. Paper Write-ups: To help make our discussions more interesting, you are expected to read each paper before class. To ensure this, before each class you must e-mail me (and the scribe for the lecture) a short write-up that addresses the question for that paper. The question for each paper is listed here. Your write-up should be approximately three paragraphs. Your e-mail should be in plain text (no attachments) and the subject should be "739: [keyword for paper]" where [keyword for paper] is the one word I used in the Lecture Schedule. Scribing: To help record the insightful comments that you all make, each lecture will have a student scribe associated with it. The scribe will be responsible for recording the general points that we discuss. The scribe will type up their notes (using this latex template) and posting these notes on the Writeup web page. Presentation: There are many interesting distributed systems papers that the class as a whole won't have time to read. To give you exposure to a few more of these papers, small groups of students will lead a lecture giving an overview of a few related papers. We will schedule these student-led lectures throughout the semester, as appropriate. You should create slides so that your talk is well organized and structured. Projects This semester, you will be doing both a warm-up project as well as a final project. The goal of the warm-up project is to give you some exposure to working with distributed systems. This warm-up project will involve setting up and using Hadoop, an open-source implementation of MapReduce and a distributed file system. You should work on the warm-up project on your own. The final project in this course has two goals. The first goal is to help you learn more about doing research in general. The second goal is to give you the opportunity to study a particular area of distributed systems in greater detail. You are expected to perform a fairly substantial research project; this involves selecting an (mostly) open problem, reading the related work, designing, implementing, and evaluating a solution, and presenting your results. People should work in groups of size two. The best project for you is the one you feel motivated to do and will learn the most from. The project is your chance to work on something new and different -- I sincerely hope that not only will you learn a lot while doing your project, but also that you will have a lot of fun! Throughout the semester, there will be a few milestones that you must meet for your main project: Initial Meeting: We will use the initial meeting to help figure out which projects would be a good match for your interests and previous experience; therefore, before this initial meeting, you should have narrowed down the list of potential projects to two or three that you are interested in. Of course, you are more than welcome to informally talk to me about potential projects before this meeting! Proposal Report: After our meeting, you should determine exactly what problem you will solve. In your project proposal, you should clearly state the problem you are planning to solve, your initial plans for approaching the problem, and your proposed methodology for evaluation. This report should contain references to at least one related background paper that you have read (but you can include more, as appropriate). Your write-up should be approximately a full page in length (or 500-1000 words). This is due Not Yet Determined. Update Meeting: Near the end of the semester, we will have a meeting to discuss your results. At this meeting, you should summarize the problem you are solving, your approach, your methodology, and your results; you should plan how you will talk about these issues, but you do not need to prepare slides. You should absolutely bring all of your experimental results (in graph or table form). At this meeting, you will receive feedback on what you need to complete before your final report. This meeting will be held around Not Yet Determined. In-class presentations: All students will give a brief overview of their projects in the last week of classes. Each group will have 20 minutes to describe their work to the rest of the class. Given this short time-frame, you will be giving only a very broad overview of your results; do not expect to distill everything that you did and learned in this talk! The time limit will be strictly enforced! More information on giving a presentation is available. Final report: Your final write-up should be similar in style, length, and content to an OS conference paper. This is clearly the most important part of demonstrating your project! More details on writing the report will be given in class. This will due Not Yet Determined.