CS 760: Machine Learning (Fall 2011)

Instructor: David Page
page@biostat.wisc.edu
Office: 6743 Medical Sciences Center (corner of Charter and University)
Office Hours: 1pm-2pm Tuesdays and 11am-Noon Fridays
Office Phone: 265-6168
TA: Quian Wan
qwan@cs.wisc.edu

Important Dates:
- Mid-term Exam: Monday, Oct 17, 11am-12:30pm
- Final Exam: Monday, Dec 5, 11am-12:30pm
- Project Due: Monday, Dec 19 by 11:59pm

Prerequisite: CS 540 or equivalent
Meeting Time and Location: 11am-12:15pm MWF, 2345 Engineering Hall
Textbook:
- Tom Mitchell (1997). Machine Learning. McGraw-Hill.
Archive of class e-mail

Course Overview

Many of the same technologies underly adaptive autonomous robots, scientific knowledge discovery, adaptive game playing and discovery from databases. This course will focus on these key underlying technologies, particularly supervised learning. The course will cover support vector machines, decision tree learners, neural network learning and Bayesian classifiers, among others. It also will address reinforcement learning and learning from relational data, including statistical relational learning and inductive logic programming. It will cover correct evaluation methodology, including case studies of methodological errors.

Course Outline

Rote Learning, Feature Vector Representation, Instance-Based Learning (UPDATED) (Mitchell Ch. 1, 2.1, 2.2, 8.1 and 8.2)
Decision Trees (UPDATED) (Mitchell Ch. 3)
Machine Learning Methodology (UPDATED) (Mitchell Ch. 5; Optional Supplements: The Case Against Accuracy Estimation for Comparing Induction Algorithms by F. Provost, T. Fawcett, and R. Kohavi, Proc. ICML-98; The Relationship Between Precision-Recall and ROC Curves by J. Davis and M. Goadrich, Proc. ICML-06)
Neural Networks (UPDATED) (Mitchell Ch. 4)
Naive Bayes (UPDATED) (Mitchell Ch. 6, supplementary background notes on probability and Bayesian Networks: 1, 2, 3)
Bayesian Network Learning (Heckerman Tutorial; Recommended: Friedman, Geiger & Goldszmidt, Machine Learning Journal 1997; Friedman, Nachman & Peer, UAI-99; Mitchell Ch. 6; additional lecture notes on Gibbs Sampling and MCMC theory [PDF])
Computational Learning Theory (UPDATED) [PDF](Mitchell Ch. 7)
EXAM: Oct 17, 11am-12:30pm
Support Vector Machines (Chris Burges's tutorial (optional))
SVM by Sequential Minimal Optimization (SMO)[PDF] (Platt's original SMO paper)
Ensemble Methods [PDF] (Dietterich, 2002)
Introduction to Rule Learning and Inductive Logic Programming (UPDATED) (Mitchell Ch. 10; for added background see De Raedt & Muggleton)
Rule Learning and Inductive Logic Programming (Mitchell Ch. 10; for added background see De Raedt & Muggleton)
- Logical Foundations
- ILP as Refinement Graph Search (Lavrac & Dzeroski, Sections 3.2, 4.1, Chapter 7)
Statistical Relational Learning
- Plate Models and Probabilistic Relational Models (Heckerman, Meek & Koller)
- Markov Logic Networks: see Lecture by Pedro Domingos on Statistical Modeling of Relational Data (Domingos & Richardson)
- View Learning in SRL: With an Application to Mammography (Davis et al., 2007)
Correlation Immune Functions (Skewing: An Efficient Alternative to Lookahead for Decision Tree Induction by Page, D. and Ray, S., IJCAI-03)
Reinforcement Learning (UPDATED) (Mitchell Ch. 13)
Genetic Algorithms (Mitchell Ch. 9)
Real-Valued Prediction Methods (Andrew Moore's Regression Notes, Smolla and Scholkopf, 2004)
Bias-Variance Tradeoff (Dietterich Lecture Notes, Domingos, 2000)
Multiple Instance Learning

Course Requirements

The grading for the course will be be based on:

Programming Assignments (2 anticipated): 15%
Exams (mid-term and final): 60%
Project: 25%

Homework Policy

The programming assignments are to be done individually. You may communicate with other class members about the problem, but please do not seek or receive help from people not in the class, and please do not share answers or code. Projects may be done in groups of up to 3 people. Programming assignments are due at the start of class on the assigned due date, and late homeworks will be penalized 10 points (out of 100) for each lecture that passes after the assigned due date. At the start of the course every student will be given 5 "free" days, each of which may be used to offset a 10-point late penalty. Free days are non-transferable, and no credit will be given for unused free days. Nevertheless, please use them sparingly because the late penalty is strictly enforced.

Homework Assignments

Assignment 0. Assigned 9/2, Due 9/9.

Assignment 1. Decision Trees and kNN. Assigned Wed, 9/14, Due Mon, 9/26. Note that this written homework does not have to be handed in; a solution set is here.

Additional Practice. Parameter Learning in Neural Nets and Bayes Nets. Solutions can be found here to Question 1, Question 2, and Question 3.

Assignment 3. Bayesian Network Learning. Assigned 9/26, Due 10/10.

Assignment 4. Support vector machines (SVM) by Platt's Sequential Minimal Optimization (SMO). Assigned 10/26, Due 11/9.

Assignment 5. Practice with Reinforcement Learning, SVMs, ILP and Markov networks. Note that this written homework does not have to be handed in; A solution set will be posted here by Wed, Nov 30.

Project

Projects are due by 11:59pm on Monday, December 19; they should be proposed by October 26 (verbal or email communication is acceptable). Projects may be done in groups of up to three people. The basis for the project grade will be your written report, which must be turned in no later than the last day of final exams. The report should be in the style of a conference paper, providing an introduction/motivation, discussion of related work, a description of your work that is detailed enough that the work could be replicated, and a conclusion. The format of the description of your work will depend on the nature of your project. If it is an implementation, then the description should make clear the algorithm(s) implemented and provide experimental results. If it is an application project, the description should say which system was used, how the data (or any other materials used) were collected, what experimental methodology was employed, and some estimate of the quality of the experimental results (e.g. a 10-fold cross-validation accuracy estimate). If it is a theoretical project, then the project description should consist of detailed definitions, theorems, and proofs.