CS 838: Big Data Systems

CS 838: Big Data Systems
Fall '16: Course Home Page

Overview, Architecture

[Architecture: Compute+Overall (B+09)] The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines , L.A. Barroso, U. Holzle, Synthesis Lectures on Computer Architecture, 2009. Chapter 1 and 2.
[Architecture: Networks: Optional (G+09)] VL2: A Scalable and Flexible Data Center Network, Greenberg et al., SIGCOMM 2009.
[Architecture: Storage (S+10)] The Hadoop Distributed File System, Schvachko et al, MSST, 2010.

Execution Engines, Resource Negotiators, Schedulers

[Execution: MR (G+04)] MapReduce Simplified Data Processing on Large Clusters, Dean and Ghemawat, OSDI, 2004.
[Execution: Stragglers (A+10)] Reining in the Outliers in Map-Reduce Clusters using Mantri, Ananthanarayanan et al, OSDI, 2010.
[ResourceNeg: YARN (V+13)] Apache Hadoop YARN: Yet Another Resource Negotiator, Vavilapalli et al, SOCC, 2013.
[ResourceNeg: Mesos: Optional (Z+11)] Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center, Zahahria et al, NSDI, 2011.
[ResourceNeg: DRF (G+11)] Dominant Resource Fairness: Fair Allocation of Multiple Resource Types, Ghodsi et al, NSDI, 2011.
[Scheduling: Packing (G+14)] Multi-Resource Packing for Cluster Schedulers, Grandl et al, SIGCOMM, 2014.
[Execution: Spark (ZC+10)] Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing, Zaharia et al, NSDI, 2012.
[Storage: Tachyon: Optional (L+15)] Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks, Li et al, SoCC, 2015.
[Execution: Tez (SS+15)] Apache Tez: A Unifying Framework for Modeling and Building Data Processing Applications, Saha et al, SIGMOD, 2015.
[Scheduling: LowLat: (O+13)] Sparrow: Distributed, Low Latency Scheduling, Ousterhout et al, SOSP, 2013.
[ResourceNeg: Borg (and Omega, Kubernetes) (V+15)] Large-scale cluster management at Google with Borg, Verma et al, Eurosys, 2015. Also read this comparison of Omega, Borg and Kubernetes.

Applications: Batch Analytics

[SQL: Hive (H+14)] Major technical advancements in apache hive, Huai et al, SIGMOD, 2014.
[SQL: Comparison Optional (F+14)] SQL-on-Hadoop: Full Circle Back to Shared-Nothing Database Architectures , A. Floratou et al, VLDB 2014.
[SQL: SparkSQL Optional (A+15)] Spark SQL: Relational Data Processing in Spark, Armburst et al, SIGMOD, 2015.
[GeoDistributed: Geode (VCB+15)] Global Analytics in the Face of Bandwidth and Regulatory Constraints, Vulimiri et al, NSDI, 2015.
[GeoDistributed: Clarinet Optional (VA+16)] Clarinet: WAN-Aware Optimization for Analytics Queries, Viswanathan et al, OSDI, 2016.

Applications: Stream Analytics

[Streaming: Storm (T+14)] Storm @Twitter , Toshniwal et al, SIGMOD, 2014.
[Streaming: Heron Optional (KB+15)] Twitter Heron: Stream Processing at Scale, Kulkarni et al, SIGMOD, 2015.
[Streaming: SparkStreaming (ZD+13)] Discretized Streams: Fault-Tolerant Streaming Computation at Scale, Zaharia et al, SOSP, 2013. Also read this introduction to Structured Streaming.
[QMS: Kafka (K+11)] Kafka:a Distributed Messaging System for Log Processing, Kreps et al, NetDB Workshop, 2011. Also read this comparison of widely used Queuing Messaging Processing Systems.
[Streaming: rStreams (L+16)] StreamScope: Continuous Reliable Distributed Processing of Big Data Streams, Lin et al, NSDI, 2016.

Applications: Graph Processing

[GraphProc:Pregel (M+10)] Pregel: A System for Large-Scale Graph Processing, Malewicz et al, SIGMOD, 2010.
[GraphProc: PowerGraph: Optional (GL+12)] PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs, Gonzalez et al, OSDI, 2012.
[Graph Storage: TAO Optional (B+13)]TAO: Facebook's Distributed Data Store for the Social Graph, Bronson et al, USENIX ATC, 2013.
[GraphProc: GraphX (G+14)] GraphX: Graph Processing in a Distributed Dataflow Framework, Gonzalez et al, OSDI, 2014.
[Streaming/Execution: Naiad (M+13)] Naiad: A Timely Dataflow System, Murray et al, SOSP, 2013.

Applications: Machine Learning

[ML: ParamServ (LA+14)] Scaling Distributed Machine Learning with the Parameter Server, Li et al, OSDI, 2014.
[ML: TensorFlow (AB+16)] TensorFlow: A System for Large-Scale Machine Learning, Abadi et al, OSDI, 2016.
[DeepLearning: MS: Optional (C+14)] Project Adam: Building an Efficient and Scalable Deep Learning Training System, Chilimbi et al, OSDI, 2014.

Potpourri: Approximation and Diagnosis

[Approx: BlinkDB (AM+13)] BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data, Agarwal et al, Eurosys, 2013.
[Compression: Succinct (AS+15)] Succinct: Enabling Queries on Compressed Data, Agarwal et al, NSDI, 2015.
[Monitoring: Pivot: Optional (M+15)] Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems, Mace et al, SOSP, 2015.