http://pages.cs.wisc.edu/~epaulson/cs764-spring12.html

Here are some footnotes to my CSx64 lecture on Big Data. There's no set order of what you should read, and it certainly isn't necessary to read everything or even anything. I just wanted to give some longer references to things I mentioned during the lecture if any of it sounded interesting.

General Overview/Jobs in the field

Jeff Hammerbacher: "Information Platforms and the Rise of the Data Scientist" This is Jeff's chapter in an O'Reilly book he edited titled 'Beautiful Data'. It's a good overview of how he gradually came to realize that they were doing something that felt different at Facebook, and that it might deserve a new name. Now, the role of a 'data scientist' is one of the hottest jobs in industry.
Information Platforms and the Rise of the Data Scientist

Steve Lohr of The New York Times: "For Today's Graduate, Just One Word: Statistics", Aug 5th 2009
"Data Science", Statistics, and "Big Data" are closely related. The Grey Lady included a story on how Statistics was turning into a hot field, and three years later, it still holds true.
http://www.nytimes.com/2009/08/06/technology/06stats.html

Big Data in Science

Moshe Vardi, editor of 'Communications of the ACM', pushed back on the notion that "Data-intensive exploration" deserved to be a new pillar of science, and disputed that even simulation/computational science was nothing more than a facet of the two main branches of science, experimental and theoretical. He titled his 1-page essay "Science Has Only Two Legs" in the September 2009 'Communications of the ACM'

Bill Howe of the University of Washington is the "In-Ferro" coiner, as near as I can tell. Bill's idea was that "Hypotheses are increasingly tested by evaluating queries over massive datasets in secondary storage - in ferro experiments - rather than relying solely on in situ, in vitro, and in silico experiments as primary means of scientific discovery." (The 'in-ferro' name comes from finding answers from magnetic disks, i.e. spinning rust) His homepage lists his papers and projects, and I think it gives a nice overview of how computing can have an impact on science, beyond just running a simulation faster.

Wired wrote an outlandish article a few years back titled "The End of Theory: The Data Deluge Makes the Scientific Method Obsolete". Like many things Wired writes, it makes over-the-top claims and predictions that can look hilariously bad a few years later. (They famously predicted in 1997 that we are watching the beginnings of a global economic boom on a scale never experienced before. and that the boom would last for 25 years. (The dot-com bust was four years later, 21 years early. And let's not even talk about 2008.)) Peter Norvig, director of research at Google (and, if you've taken AI, likely your textbook author) was a subject of the 'End of Theory' piece and felt compelled to to write a rebuttal. While I wouldn't take this Wired piece as gospel, it is still a fun read, and there are a dozen or so vignettes in the side bar that are much more reality-based and worth reading.

Speaking of Peter Norvig, he with some other Googlers/Washington Faculty wrote an essay titled The Unreasonable Effectiveness of Data for IEEE. It looks into how using large amounts of data can help attack some difficult problems in language processing and extracting data from the web. It's just a few pages long and has no equations at all and is a nice gentle introduction to the subject.

Big Data Science Projects

• An article about the Sloan Digital Sky Survey, and how it is being used to democratize science.
• The Large Synoptic Survey Telescopeis under construction in Chile. It will photograph the entire sky every 3 days, and produce about a petabyte of image data a year.
• NEON, the National Ecological Observatory Network is a project to "collect data across the United States on the impacts of climate change, land use change and invasive species on natural resources and biodiversity."
• The Earth Cube Project is a repository for geological data. The UW-Madison hopes to be a part of this one.

Interesting Applications

• Learning Analytics. Can we use 'big data' techniques to help identify students who are struggling? People at the UW-Madison are interested in deploying something like this, to identify students who are struggling early so the professor can intervene and to help create "data-driven" approaches to evaluating different pedagogical approaches.
• Ushahidi is a collaboration platform that most often springs up in times of crisis. There are some amazing stories of how Ushahidi helped save lives by connecting translators in the US with rescue crews in Haiti.
• Big Data in the Humanities. It's not all physicists and biologists working with "big data".
• Big Data and elections. All of the candidates in 2012 will be using behavioral testing to help inform their communications with voters. Most of the experiments in this article were done "off-line", i.e. in the mail or over the phone, but the Internet makes it possible to do the same experiments in real time and to be constantly experimenting.

Paper references

• This was the paper most of the results I talked about were initially presented: A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker, "A comparison of approaches to large-scale data analysis" in SIGMOD '09: Proceedings of the 35th SIGMOD international conference on Management of data, New York, NY, USA, 2009, pp. 165-178.
• This was the revised version we did for the Communications of the ACM: Michael Stonebraker, Daniel Abadi, David J. DeWitt, Sam Madden, Erik Paulson, Andrew Pavlo, and Alexander Rasin. (2010) MapReduce and Parallel DBMSs: Friends or Foes? Communications of the ACM, 53(1):64-71.
• Finally, this is the response paper from Google: Jeffrey Dean and Sanjay Ghemawat. (2010) MapReduce: A Flexible Data Processing Tool. Communications of the ACM, 53(1):72-77.