CS 838: Big Data Systems

CS 838: Big Data Systems
Fall '15: Course Home Page

Overview, Architecture

[Architecture: Compute+Overall (B+09)] The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines , L.A. Barroso, U. Holzle, Synthesis Lectures on Computer Architecture, 2009. Chapter 1.
[Overview: AboveTheClouds (A+09)] Above the Clouds: A Berkeley View of Cloud Computing, M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R.H. Katz, A. Konwinski, G. Lee, D.A. Patterson, A. Rabkin, I. Stoica, M. Zaharia, Tech. Rep. UCB/EECS-2009-28, Feb 10, 2009.
[Architecture: Networks (S+15)] Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network, Singh et al., SIGCOMM 2015.
[Architecture: Storage (S+10)] The Hadoop Distributed File System, Schvachko et al, MSST, 2010.

Execution Engines, Resource Negotiators, Schedulers

[Execution: MR (G+04)] MapReduce Simplified Data Processing on Large Clusters, Dean and Ghemawat, OSDI, 2004.
[Execution: Stragglers: Optional (A+10)] Reining in the Outliers in Map-Reduce Clusters using Mantri, Ananthanarayanan et al, OSDI, 2010.
[ResourceNeg: YARN (V+13)] Apache Hadoop YARN: Yet Another Resource Negotiator, Vavilapalli et al, SOCC, 2013.
[Scheduling: Packing: Optional (G+14)] Multi-Resource Packing for Cluster Schedulers, Grandl et al, SIGCOMM, 2014.
[Execution: Spark (ZC+10)] Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing, Zaharia et al, NSDI, 2012.
[Execution: Tez (SS+15)] Apache Tez: A Unifying Framework for Modeling and Building Data Processing Applications, Saha et al, SIGMOD, 2015.
[Execution: REEF: Optional (C+15)] REEF: Retainable Evaluator Execution Framework, Chun et al, SIGMOD, 2015.
[Scheduling: LowLat: (O+13)] Sparrow: Distributed, Low Latency Scheduling, Ousterhout et al, SOSP, 2013.
[Scheduling: SharedState: Optional (S+13)] Omega: flexible, scalable schedulers for large compute clusters, Schwarzkopf et al, Eurosys, 2013.
[ResourceNeg: Mesos: Optional (Z+11)] Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center, Zahahria et al, NSDI, 2011.
[Storage: Tachyon (L+15)] Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks, Li et al, SoCC, 2015.

Applications: SQL-based, Stream Analytics

[AppFramework: Hive (H+14)] Major technical advancements in apache hive, Huai et al, SIGMOD, 2014.
[AppFramework: SparkSQL (A+15)] Spark SQL: Relational Data Processing in Spark, Armburst et al, SIGMOD, 2015.
[AppFramework: Impala (K+15)] Impala: A Modern, Open-Source SQL Engine for Hadoop, Kornacker et al, CIDR 2015.
[AppFramework: Presto: Optional (Cb+13)] Presto: Interacting with petabytes of data at FacebookP , blog by L. Chang, 2013.
[AppFramework: Comparison (F+14)] SQL-on-Hadoop: Full Circle Back to Shared-Nothing Database Architectures , A. Floratou et al, VLDB 2014.
[Streaming: SparkStreaming (ZD+13)] Discretized Streams: Fault-Tolerant Streaming Computation at Scale, Zaharia et al, SOSP, 2013.
[Streaming: MROnline (C+10): Optional] MapReduce Online, T. Condie, N. Conway, P. Alvaro, J.M. Hellerstein, K. Elmeleegy, R. Sears, NSDI 2010.
[Streaming: Storm (T+14)] Storm @Twitter , Toshniwal et al, SIGMOD, 2014.
[Streaming: Heron (KB+15)] Twitter Heron: Stream Processing at Scale, Kulkarni et al, SIGMOD, 2015.

Applications: Graph Processing, Machine Learning

[GraphProc: GraphX (G+14)] GraphX: Graph Processing in a Distributed Dataflow Framework, Gonzalez et al, OSDI, 2014.
[GraphProc: PowerGraph: Optional (GL+12)] Powergraph: Distributed graph-parallel computation on natural graphs, Gonzalez et al, OSDI, 2012.
[GraphProc: Comparison: (HD+14)] An Experimental Comparison of Pregel-like Graph Processing Systems, Han et al, CIDR, 2013.
[ML: MLBase: (KT+13)] MLbase: A Distributed Machine-learning System, Kraska et al, CIDR, 2013.
[ML: MLI: (ST+13): Optional] MLI: An API for Distributed Machine Learning, Sparks et al, ICDM, 2013.
[ML: ParamServ (LA+14)] Scaling Distributed Machine Learning with the Parameter Server, Li et al, OSDI, 2014.
[DeepLearning: MS (C+14)] Project Adam: Building an Efficient and Scalable Deep Learning Training System, Chilimbi et al, OSDI, 2014.
[DeepLearning: NetFlix: Optional (N+15)] Distributed Neural Networks with GPUs in the AWS Cloud.

Potpourri: Geo-distributed setting, Approximation, and Diagnosis

[Geo: Geode (VCB+15)] Global Analytics in the Face of Bandwidth and Regulatory Constraints, Vulimiri et al, NSDI, 2015.
[Geo: WANalytics (VC+15): Optional] WANalytics: Analytics for a geo-distributed data-intensive world, Vulimiri et al, CIDR, 2015.
[Geo: GDA (PA+15): Optional] Low-Latency Analytics of Geo-Distributed Data in the Wide Area, Pu et al, SIGCOMM, 2015.
[Approx: BlinkDB (AM+13)] BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data, Agarwal et al, Eurosys, 2013.
[Monitoring: Pivot (M+15)] Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems, Mace et al, SOSP, 2015.