CS 638 and CS 838 - Building Deep Neural Networks

Instructor: Jude Shavlik

Office:      6393 CS Building 
Email:       shavlik@cs.wisc.edu 
Office Hours: Tuesday 2-4pm (also for undergraduate advising 3-4 PM). Also by appointment (send email). 

Teaching Assistant: Yuting Liu

Office:        1308 CS Building 
Email:         yuting@cs.wisc.edu 
Office Hours:  Wednesday 3:30-4:30pm.  Also by appointment (send email). 

Additional Information

• Prerequisite: CS 540 or CS 760
• Meeting schedule: Tuesday 5:30-8:20pm (first class day is January 17 and last class day is May 2)
• Location: 1240 Computer Sciences
• Textbook: Deep Learning by Goodfellow, Bengio, and Courville, MIT Press, 2016 (free version available online).
• Moodle class page for course; you need to turn in lab reports via Moodle
• Pizza discussion page

Table of Contents

• Course Overview and Requirements
• Homeworks
• Reading Assignments
• Lecture Slides
• Java On-line Textbook | Java 1.7 Manual| Java 1.8 Manual
• The `Agent World' Simulator (useful for reinforcement learning)
• Available Software and Free 'Cloud' Accounts for Students
• Some Relevant Articles
• Graduate AI Courses at Wisconsin

Course Overview and Requirements

The course will be graded on the conventional (A-F) system.

Homeworks

Use Moodle to turn in your code and reports. It is fine to discuss design, data structures, and algorithms with others in the class, but only members of the same two or four-person group can share actually code unless explicitly authorized by the course's instructor.

• Introduction to Deep Learning textbook by Goodfellow et al. (linked above).
• Review neural-network chapters from cs540 or cs760.

Lecture Notes

I'll post some DRAFT lecture notes before class, BUT I'll post edited versions after the class. You might want to print them out just before class, using something like three slides to page and leaving half of each the printed page empty for writing notes during class.
• Presented Week 1: course overview | lecture17, lecture18, lecture19, and lecture20 (these are from my cs540)
• Presented Week 3: reinforcement learning (continued Week 5)
• Presented Week 4: Initializing weights | Lab 3
• Presented Week 5: Knowledge-Based Artificial Neural Networks
• Presented Week 7: Tips for Lab 3 | Methods for Weight Update in Neural Networks presented by Yujia Bao (matlab code for generating the plots, zipped) | Statistical Analyis of What BP Should Optimize (old cs760 lecture notes)
• Presented Week 8: More Tips for Lab 3 | Using Finite Differencing to Check your Gradients
• Presented Week 9: Class Projects | Batch Renormalization presented by Glenn Fung
• Presented Week 10: More Lab 3 Comments | LSTM etc by Akshay Sood | Some Industrial Deep Learning Projects by Luisa Polania Cabrera
• Presented Week 11: Keras for Lab 3 | Deep Learning and NLP by Sidharth Mudgal Sunil Kumar
• Presented Week 12: Generative Adversarial Networks (GANs) presented by Gabe Ferns & Max Horowitz-Gelb and
• Presented Week 13: Google's Tensor Processing Unit (TPU) a version of the talk presented by Andy Phelps of the Madison Google Office
• Image datasets and Java/Kera code available here.

• Quiz (perceptrons and backpropagation; convolution, max pooling, and drop out): TBA (only for students, if any, with insufficient project progress by April 14)
  ONE (8.5x11) page of notes (you can write on both sides of sheet) and a calculator allowed.

• MXNet (Amazon) (send me better web pages for these if you know of them)
• Paddle (Baidu)
• Torch (Facebook)
• Tensor Flow (Google)
• Power AI (IBM)
• CNTK (Microsoft)
• Caffe (U-Berkeley)
• Theano (U-Montreal)
• Keras (Deep Learning Python library for Theano and TensorFlow)
• Some (Python-related) Suggestions for Learning about Deep Learning
• Some ML Datasets
• Amazon student accounts (ditto; for simplicity, we'll pick one option for use by the entire class, with exceptions allowed if strong reasons for doing so)
• Google student accounts
• IBM student accounts
• Microsoft student accounts

• Deep Learning Resources
• Deep Learning for Java
• Some Interesting Observations at the 2016 NIPS Conferences
• Fifteen Deep Learning tutorials
• "Deep Learning in Neural Networks: An Overview," by Juergen Schmidhuber
• The Great AI Awakening (NY Times article about Google)
• Nice page on DL tutorials, tasks, and testbeds
• The 'ML News' mailing list: ml-news+subscribe@googlegroups.com
• Tutorial on Generative Adversarial Networks
• Google Releases a 3-D World for Smarter AI Agents
• COCO, an Annotated Image Dataset from Microsoft
• YouTube-8M, a large-scale labeled video dataset that consists of 8 million YouTube videos
• The Association for the Advancement of Artificial Intelligence (AAAI)
• AI Magazine (old issues available on-line for free)