HAMLET
Human, Animal, and Machine Learning: Experiment and Theory
To start the semester off with a bang, we will engage a cross-disciplinary discussion of the "replication crisis" in psychology. Over the past few years some high-profile scandals, statistical exposees of impossible patterns of published results, and some firings, all in disciplines related to psychology writ large, led to a large-scale effort to replicate 100 studies sampled from relatively high-profile journals. The results were published a few weeks ago in Science (http://www.sciencemag.org/content/349/6251/aac4716). They don't look great---according to the authors, only about 30% of the studies replicated the original effect.
There has been a lot of discussion about what this means. Is the discipline full of frauds? Is there something we don't understand about commonly-used statistical methods in the discipline? Are there culture-of-science issues that lead to inflated estimates of effect sizes? Is the replication study itself inherently flawed? Are there better methods we should be borrowing from other disciplines? How well do other disciplines do in replication? And so on---see below for a list of links to various blog posts and media articles on the topic.
I will provide a very brief overview of the issue. We are then fortunate to have three outstanding faculty from different disciplines who will provide their perspective in a brief and informal set of comments, possibly with slides: Elliot Sober from the Philosophy Department, whose expertise lies in philosophy of science; Shawn Green from the Psychology Department, who has pioneered hierarchical Bayesian approaches to perceptual learning in psychology; and Peter Steiner, a statistician from Educational Psychology who works on state of the art methods of causal inference in that discipline. The aim is to use these perspectives to kick-start a general discussion of how this kind of science ought to be done.
In HAMLET this week we will be hearing from Rob Nowak from ECE, who will be speaking about NEXT, a new online system for efficiently estimating the information structures that underlie human judgments in a wide range of domains. NEXT is a developing collaborative project that can be used to find the top k amongst n items, to develop full rankings, and to estimate low-dimensional embeddings amongt items, all from pairwise human judgments sourced from the crowd. These ideas have been used to develop the "beer mapper" (see this Wired article: http://www.wired.co.uk/news/archive/2013-05/21/beer-mapper) and to search for the best logo for the LUCID graduate training program. Rob will unveil the top-ranked entries! NEXT will also allow for comparison of different algorithms for efficient estimation from the crowd. Rob will talk about some of the theory behind the system, some of its uses, and current and planned research directions.
Cognitive models of human learning often assume that learners are rational: in a given learning setting they combine prior knowledge and evidence so as to make optimal probabilistic inferences for new observations. This view can be hard to reconcile with the daily news, in which large groups of people appear to hold opposing beliefs on what ought to be matters of fact: whether GMO foods are safe, whether vaccines cause Autism, whether the climate is changing, whether tax cuts stimulate job growth, and so on. If people are rational learners, why do they come to form very different beliefs in these cases? I will describe empirical and computational work exploring one hypothesis: maybe incorrect beliefs form because learners in the real world are presented with many conflicting sources of information and must decide which sources to "trust." The behavioral studies illustrate two "paradoxes" that are difficult to understand from rational approaches to learning, and I am very interested to know whether they make any sense to people in the machine learning community. The computational work shows that, when simulated groups of learners interact by sharing labels with one another, they often break into mutually distrustful clusters with differing beliefs, providing a possible mechanism for understanding how incorrect beliefs can persist in communities of learners.
Tim Rogers (ttrogers@wisc.edu), Jerry Zhu (jerryzhu@cs.wisc.edu)