CS 744: Big Data Systems

CS 744: Big Data Systems
Fall '17: Course Home Page

[ Home | Reading List | Schedule | Assignments | Piazza ]

Assignments and projects are to be done in groups of 3 students. There will be three assignments and one course project. All the assignments are mandatory for each group; If you are in traditional MS or PhD program in Computer Science, you also need to do a research project in addition to the assignments. The assignments are well-defined and focus on providing hands-on experience with various big data systems, where as the projects tend to be more open-ended and are important for those who wish to pursue research in this field.

Cloud Resources

You would be required to use resources in the cloud for all experiments. We will be using CloudLab for that.

Assignments

Each group is expected to turn in a PDF writeup answering the questions posed in the assignment. The tentative list of assignments is as follows:

Assignment 1 - Fun with MR, Tez, and Spark : In this assignment, you will run canonical benchmark workloads using Apache Hive atop Map-Reduce and Tez. You will study the differences in the two systems in detail. You will also learn how to write simple MapReduce and Spark applications. Released Sep 7. Due ~~Sep 28,~~ Oct 1.
Assignment 2 - Even More Fun with Structured Streaming, Storm and Flink : In this assignment, you will learn how to write end-to-end streaming applications using Structured Streaming. You will also develop and run a real-time tweet processing streaming application on Apache Storm and Apache Flink. Released Oct 13. Due Oct 29.
Assignment 3 - Supreme Fun with TensorFlow and GraphX : In this assignment, will learn how to write native applications on TensorFlow and GraphX. Released Nov 14. Due Dec 3.

Projects

A list of course projects to consider will be released early into the semester. Each project in the list will contain a short description of the problem, the questions that you will most likely answer and a sketch of initial steps to take. Do not panic if you are not able to comprehend the description and goals upon first reading. The related work will be covered in class as the semester progresses. The purpose of releasing the list early is to give sufficient time for background reading.

The schedule for the project deliverables is as follows:

Project Milestone

Due Date

Proposal

Thursday 10/05

Related Work Report

Friday 10/13

Initial Results

Friday 11/03

Final Report

Tuesday 12/12

Project Milestone	Due Date
Proposal	Thursday 10/05
Related Work Report	Friday 10/13
Initial Results	Friday 11/03
Final Report	Tuesday 12/12