Books/Courses: no textbooks are required. Please get familiar with the basic analysis tools in the books listed in the Prerequisite. The following books are additional ones, part of which may be used for or complementary to the lectures.

• [T21] Matus Telgarsky, Deep learning theory lecture note, 2021-10-27 v0.0-e7150f2d.
• [A+21] Sanjeev Arora et al., Theory of Deep learning book draft, 2021.

Some course materials from the following course will be used:

• [G21] Quanquan Gu. CS269 Foundations of Deep Learning, Spring 2021.

We thank the authors for providing these excellent materials!

Papers:

• [ZBHRV17] Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. "Understanding deep learning requires rethinking generalization." In International Conference on Learning Representations. 2017.
• [SHNGS18] Soudry, Daniel, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. "The implicit bias of gradient descent on separable data." The Journal of Machine Learning Research 19, no. 1 (2018): 2822-2878.
• [GLSS18] Gunasekar, Suriya, Jason Lee, Daniel Soudry, and Nathan Srebro. "Characterizing implicit bias in terms of optimization geometry." In International Conference on Machine Learning, pp. 1832-1841. PMLR, 2018.
• [NLGSSS19] Nacson, Mor Shpigel, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. "Convergence of gradient descent on separable data." In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3420-3428. PMLR, 2019.
• [DZPS18] Du, Simon S., Xiyu Zhai, Barnabas Poczos, and Aarti Singh. "Gradient Descent Provably Optimizes Over-parameterized Neural Networks." In International Conference on Learning Representations. 2018.
• [CB18] Chizat, Lénaïc, and Francis Bach. "On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport." Advances in Neural Information Processing Systems 31 (2018): 3036-3046.
• [JGH18] Jacot, Arthur, Franck Gabriel, and Clément Hongler. "Neural tangent kernel: convergence and generalization in neural networks." Advances in Neural Information Processing Systems. 2018.
• [FDZ19] Fang, Cong, Hanze Dong, and Tong Zhang. "Over parameterized two-level neural networks can learn near optimal feature representations." arXiv preprint arXiv:1910.11508 (2019).
• [MMN18] Mei, Song, Andrea Montanari, and Phan-Minh Nguyen. "A mean field view of the landscape of two-layer neural networks." Proceedings of the National Academy of Sciences 115, no. 33 (2018): E7665-E7671.
• [GL20] Garg, Siddhant, and Yingyu Liang. "Functional regularization for representation learning: A unified theoretical perspective." Advances in Neural Information Processing Systems 33 (2020): 17187-17199.
• [AKKPS19] Arora, Sanjeev, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. "A theoretical analysis of contrastive unsupervised representation learning." In 36th International Conference on Machine Learning, ICML 2019, pp. 9904-9923. International Machine Learning Society (IMLS), 2019.
• [SCLRWLJ23] Shi Zhenmei, Jiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, Somesh Jha. "The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning." In International Conference on Learning Representations. 2023.
• [BR88] Blum, Avrim L., and Ronald L. Rivest. "Training a 3-node neural network is NP-complete." Neural Networks 5, no. 1 (1992): 117-127. Conference version appeared in Advances in neural information processing systems 1 (1988).
• [SWL22] Shi, Zhenmei, Junyi Wei, and Yingyu Liang. "A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Features." In International Conference on Learning Representations. 2022.