CS 839-7 Probability and Learning in High Dimension (Spring 2022)

1. Basic Info
2. Course Overview
3. Grading
4. Lecture Notes
5. Homework and Projects
6. Texts and References
7. Academic Policies

1. Basic Info

Lectures:
Monday and Wednesday 1:00-2:15pm, CS Building 1325

Instructor:
Yudong Chen (yudong.chen at wisc dot edu, CS Building 5373)
Office hours: Wednesday 2:30-3:30pm via Zoom (zoom link can be found on Canvas)

Prerequisites:
There is no formal prerequisite. Students should have a phd level of mathematical maturity, including a background in basic linear algebra, probability and algorithms. Prior exposure to machine learning, statistical inference, stochastic processes and convex/continous optimization is helpful, but not required.

Websites and communication:

• • Piazza: For discussion and course announcements. Sign up for this course on Piazza using this link.
• • Canvas: We use Canvas for posting course materials.

2. Course Overview

This is a fast-paced course on the probability and statistical tools for high-dimensional data analysis. In particular, we will develop technique for analyzing the performance of an algorithm, as wells for understanding the fundamental limits of a problem. Focus will be on the high-dimenisonal problems that possess hiden low-dimensional structuers, and on non-asymptotic anaysis that characterizes the interaction between sample complexity, problem dimension and other structural parameters. 

A main theme of this course is the demonstrate the power of probablistic tools in the study of machine learning and statistical problems. We will show that

A few tools can take us very far

... in analyzing a broad range of problems and algorithms.

Tentative list of topics:

• • Matrix concentration
• • Spectral methods 
• • Convex relaxation methods
  
• • Structured matrix estimation
• • Tensor decomposition
• • Randomized linear algebra
• • Nonparametric statistical estimation
• • Reinforcement learning and sample complexity
• • Statistical methods based on non-convex optimization
• • Information theoretic lower bounds
• • Uniform laws and localization
• • Overparametrization and double descent

Relation to MATH 888: There will be some (~20%) overlap with MATH 888 (e.g., on basic concentration inequalities). This course will otherwise cover a different set of topics, problems and applications, complementrary to what's taught in MATH 888.

3. Grading

Your final grade will be based on the following:

• • 10%: Scribing (instructions), participation in class, Piazza and course evaluation
• • 40%: Homework
• • 50%: Final Report

4. Lecture Notes

Each studen is expected to scribe one lecture.

• • Sign up here (you need to use your Wisc google account to access it). Pick your date based on your other constraints and not based on topic. Topics may change. 
• • Template for scribing can be downloaded this link to scribe template. (For some examples of what scribe notes look like, see here.)
• • A first draft of your scribe notes is due 72 hours after the lecture and should be emailed to the instructor. Submissions must include a compiled PDF, the LaTeX source, and necessary figures. The instructor may request further changes to the draft.
• • The notes will be posted on Canvas.

Note: If you decide to drop the course before your scribe date, inform the instructor as soon as possible. If you are on the waitlist, please do not sign up for scribing until you have been allowed to enroll.

Below are the scribed lecture notes:

• • Lecture 1: Introduction and Matrix Bernstein Inequality
• • Lecture 2: Non-parametric Bradley-Terry Model
• • Lecture 3: Matrix Concentration I
• • Lecture 4: Matrix Concentration II
• • Lecture 5: Kernel Methods and Random Features
• • Lecture 6: Spectral Algorithms I
• • Lecture 7: Spectral Algorithms II
• • Lecture 8: Convex Relaxation and Community Detection I
• • Lecture 9: Convex Relaxation and Community Detection II
• • Lecture 10: Max Norm and Nuclear Norm Relaxations
• • Lecture 11: Convex Relaxation and Community Detection: Exact Recovery
• • Lecture 12: Lipschitz Concentration and Gaussian Comparison Inequalities
• • Lecture 13: Random Process and Metric Entropy
• • Lecture 14: Random Processes: Chaining and Additional Tools
• • Lecture 15: Statistical Learning
• • Lecture 16–17: Non-parametric Least Squares
• • Lecture 18: Basics of Markov Decision Processes
• • Lecture 19–20: Sample Complexity of Reinforcement Learning
• • Lecture 21: Reinforcement Learning for Linear MDPs I
• • Lecture 22: Reinforcement Learning for Linear MDPs II
• • Lecture 23: Reinforcement Learning for Linear MDPs III
• • Lecture 24: Randomized Numerical Linear Algebra I
• • Lecture 25: Randomized Numerical Linear Algebra II
• • Lecture 26: Randomized Numerical Linear Algebra III

5. Homework and Projects

There will be approximately 3 homework assignments. You are encourage to discuss and work together on the homework. However, you must write up your homework alone, AND acknowledge those with whom you discussed with. You must also cite any resources which helped you obtain your solution.

There will also be a final project, to be completed individually or in groups of two. The project can be any of the following:

• • Literature review: Critical summary of one or several papers related to the topics studied.
• • Original research: It can be either theoretic or experimental (ideally a mix of the two). 

We particularly welcome projects that may be extended for submission to a peer-reviewed journal or conference (e.g., MOR/AoS/T-IT/COLT/ICML/NeurIPS/ICLR). Project topics must be approved by the instructor.

Detailed project instruction can be found here.

6. Texts and References

We recommend the following books and notes, but will not follow them closely.

• • High-Dimensional Statistics: A Non-Asymptotic Viewpoint, Martin J. Wainwright, Cambridge University Press, 2019.
  • High-Dimensional Probability: An Introduction with Applications in Data Science, Roman Vershynin, Cambridge University Press, 2018.
• • High-Dimensional Data Analysis with Low-Dimensional Models: Principles, Computations, and Applications, John Wright, Yi Ma, 2022.
• • Statistical Foundations of Data Science, Jianqing Fan, Runze Li, Cun-Hui Zhang, Hui Zou, 2020.

The following references also contain topics relevent to this course.

• • Lecture Notes for ORIE 7790: High Dimensional Probability and Statiscs, Yudong Chen, 2020.
• • Lecture Notes for Statistics 311/Electrical Engineering 377: Information Theory and Statiscs, John Duchi, 2019.
• • Lecture Notes for ELE 520: Mathematics of Data Science, Yuxin Chen, 2020.
• • Probability in High Dimension, Ramon van Handel, 2016.
• • Harnessing Structures in Big Data via Guaranteed Low-rank Matrix Estimation, Yudong Chen, Yuejie Chi, 2018.
• • Nonconvex Optimization Meets Low-rank Matrix Factorization: An Overview, Yuejie Chi, Yue M. Lu, Yuxin Chen, 2019
• • Convex Relaxation Methods for Community Detection, Xiaodong Li, Yudong Chen, Jiaming Xu, 2018.
• • An Introduction to Matrix Concentration Inequalities, Joel Tropp, Foundations and Trends in Machine Learning, 2015.
• • Graphical Models, Exponential Families, and Variational Inference, Martin Wainwright, and Michael Jordan, Foundations and Trends in Machine Learning, 2008.
• • Introduction to the Non-Asymptotic Analysis of Random Matrices, Roman Vershynin, Compressed Sensing: Theory and Applications, 2010.

7. Academic Policies

COVID Policy

Students of the class are expected to comply with the University’s current COVID rules and policies (see in particular the FAQ).

Students who do not comply with these rules can be asked to leave the classroom, and students who repeatedly fail to comply will be referred to the Office of Student Conduct and Community Standards. Any student who requires an exemption to current policies must contact the McBurney Office, as instructors do not have the authority to grant such exceptions.

Academic Conduct

You are encouraged to discuss with your peers or the instructors ideas, approaches and techniques broadly. However, all examinations, programming assignments, and written homeworks must be written up individually. For example, code for programming assignments must not be developed in groups, nor should code be shared. Make sure you work through all problems yourself, and that your final write-up is your own. If you feel your peer discussions are too deep for comfort, declare it in the homework solution: “I discussed with X,Y,Z the following specific ideas: A, B, C; therefore our solutions may have similarities on D, E, F…”.

You may use books or legit online resources to help solve homework problems, but you must always credit all such sources in your writeup and you must never copy material verbatim.

Academic integrity issues will be dealt with in accordance with University procedures (see the UW-Madison Academic Misconduct Page)

If you have any questions about this policy, please do not hesitate to contact the instructor.

Accommodations for Students with Disability

The University of Wisconsin-Madison supports the right of all enrolled students to a full and equal educational opportunity. The Americans with Disabilities Act (ADA), Wisconsin State Statute (36.12), and UW-Madison policy (Faculty Document 1071) require that students with disabilities be reasonably accommodated in instruction and campus life. Reasonable accommodations for students with disabilities is a shared faculty and student responsibility. Students are expected to inform the instructors of their need for instructional accommodations by the end of the third week of the semester, or as soon as possible after a disability has been incurred or recognized. The instructors will work either directly with the student or in coordination with the McBurney Center to identify and provide reasonable instructional accommodations. Disability information, including instructional accommodations as part of a student’s educational record, is confidential and protected under FERPA. (See: McBurney Disability Resource Center)

Diversity

Respect for Diversity: It is the intent of the instructors that students from all diverse backgrounds and perspectives be well served by this course, that students’ learning needs be addressed both in and out of class, and that the diversity that students bring to this class be viewed as a resource, strength and benefit. It is our intent to present materials and activities that are respectful of diversity: gender, sexuality, disability, age, socioeconomic status, ethnicity, race, and culture. Your suggestions are encouraged and appreciated. Please let us know ways to improve the effectiveness of the course for you personally or for other students or student groups. In addition, if any of our class meetings conflict with your religious events, please let us know so that we can make arrangements for you.

Please, commit to helping create a climate where we treat everyone with dignity and respect. Listening to different viewpoints and approaches enriches our experience, and it is up to us to be sure others feel safe to contribute. Creating an environment where we are all comfortable learning is everyone’s job: offer support and seek help from others if you need it, not only in class but also outside class while working with classmates.