# Summary

M4, M5,
P2,
Q5, Q6, Q7, Q8,

# Lectures

Blank Slides: Part 1, Part 2, Part 3, Part 4,
Blank Slides (with blank pages for quiz questions): Part 1, Part 2, Part 3, Part 4,
Blank Slides with Quiz Questions: Part 1, Part 2, Part 3, Part 4,
Annotated Slides: Part 1, Part 2, Part 3, Part 4,
PDF.



Image by Vishal Arora via medium

# Other Materials

Lecture 5 Part 1 (Support Vector Machines): Link
Lecture 5 Part 2 (Subgradient Descent): Link
Lecture 5 Part 3 (Kernel Trick): Link
Lecture 6 Part 1 (Decision Tree): Link
Lecture 6 Part 2 (Random Forrest): Link
Lecture 6 Part 3 (Nearest Neighbor): Link
Lecture 7 Part 1 (Convolution): Link
Lecture 7 Part 2 (Gradient Filters): Link
Lecture 7 Part 3 (Computer Vision): Link
Lecture 8 Part 1 (Computer Vision): Link
Lecture 8 Part 2 (Viola Jones): Link
Lecture 8 Part 3 (Convolutional Neural Net): Link

Support Vector Machine: Link
RBF Kernel SVM Demo: Link

Decision Tree: Link
Random Forrest Demo: Link

K Nearest Neighbor: Link
Map of Manhattan: Link
Voronoi Diagram: Link
KD Tree: Link

Image Filter: Link
Canny Edge Detection: Link
SIFT: PDF
HOG: PDF
Conv Net on MNIST: Link
Conv Net Vis: Link
LeNet: PDF, Link
Google Inception Net: PDF
CNN Architectures: Link
Image to Image: Link
Image segmentation: Link
Image colorization: Link, Link 
Image Reconstruction: Link
Style Transfer: Link
Move Mirror: Link
Pose Estimation: Link
YOLO Attack: YouTube


How to find the margin expression for SVM? Link
Why does the kernel trick work? Link
Example (Quiz): Compute SVM classifier Link
Example (Quiz): Kernel SVM for XOR operator Link
Example (Quiz): Kernel matrix to feature vector Link
Example (Quiz): Entropy computation Link
Example (Quiz): Decision tree for implication operator Link
Example (Quiz): Three nearest neighbor Link
How to find the HOG features? Link
How to count the number of weights for training for a convolutional neural network (LeNet)? Link
Example (Quiz): How to find the 2D convolution between two matrices? Link
Example (Homework): How to find a discrete approximate Gausian filter? Link

# Kernel Demo

Data type: Circle Side-x Side-y , Count:
Kernel type: Quadratic Square-x Square-y
Kernel:
Plane:

# Keywords and Notations

SVM classifier: ${\hat{y}}_i=1_{\{w^{\mathrm{\top }}{x}_i+b\ge 0\}}$.
Hard margin, original max-margin formulation: ${\displaystyle \underset{w}{max}{\displaystyle \frac{2}{\sqrt{w^{\mathrm{\top }}w}}}}$ such that $w^{\mathrm{\top }}{x}_i+b\le -1$ if $y_i=0$ and $w^{\mathrm{\top }}{x}_i+b\ge 1$ if $y_i=1$.
Hard margin, simplified formulation: ${\displaystyle \underset{w}{min}{\displaystyle \frac{1}{2}}{w}^{\mathrm{\top }}w}$ such that $(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\ge 1$.
Soft margin, original max-margin formulation: ${\displaystyle \underset{w}{min}{\displaystyle \frac{1}{2}}{w}^{\mathrm{\top }}w+{\displaystyle \frac{1}{\lambda }}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{\xi }_i}}$ such that $(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\ge 1-\xi ,\xi \ge 0$, where ${\xi }_i$ is the slack variable for instance $i$, $\lambda$ is the regularization parameter.
Soft margin, simplified formulation: ${\displaystyle \underset{w}{min}{\displaystyle \frac{\lambda }{2}}{w}^{\mathrm{\top }}w+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{\displaystyle max\{0,1-(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\}}}}$
Subgradient descent formula: $w=(1-\lambda )w-\alpha (2y_i-1)1_{\{(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\ge 1\}}{x}_i$.

Kernel SVM classifier: ${\hat{y}}_i=1_{\{w^{\mathrm{\top }}\phi \left(x_i\right)+b\ge 0\}}$, where $\phi$ is the feature map.
Kernal Gram matrix: $K_{i{i}^{\prime }}=\phi {\left(x_i\right)}^{\mathrm{\top }}\phi \left(x_{i^{\prime }}\right)$.
Quadratic Kernel: $K_{i{i}^{\prime }}={(x_{i^{\mathrm{\top }}}{x}_{i^{\prime }}+1)}^2$ has feature representation $\phi \left(x_i\right)=(x_{i1}^2,x_{i2}^2,\sqrt{2}{x}_{i1}{x}_{i2},\sqrt{2}{x}_{i1},\sqrt{2}{x}_{i2},1)$.
Gaussian RBF Kernel: $K_{i{i}^{\prime }}=\exp (-{\displaystyle \frac{1}{2{\sigma }^2}}{(x_i-x_{i^{\prime }})}^{\mathrm{\top }}(x_i-x_{i^{\prime }}))$ has infinite-dimensional feature representation, where ${\sigma }^2$ is the variance parameter.

Entropy: $H\left(Y\right)=-{\displaystyle \sum _{y=1}^K{p}_y{\log }_2\left(p_y\right)}$, where $K$ is the number of classes (number of possible labels), $p_y$ is the fraction of data points with label $y$.
Conditional entropy: $H\left(Y|X\right)=-{\displaystyle \sum _{x=1}^{K_X}{p}_x{\displaystyle \sum _{y=1}^K{p}_{y|x}{\log }_2\left(p_{y|x}\right)}}$, where $K_X$ is the number of possible values of feature, $p_x$ is the fraction of data points with feature $x$, $p_{y|x}$ is the fraction of data points with label $y$ among the ones with feature $x$.
Information gain, for feature $j$: $I\left(Y|X_j\right)=H\left(Y\right)-H\left(Y|X_j\right)$.

Decision stump classifier: ${\hat{y}}_i=1_{\{x_{ij}\ge t_j\}}$, where $t_j$ is the threshold for feature $j$.
Feature selection: $j^{\star }={\mathrm{argmax}}_{j}I\left(Y|X_j\right)$.

Convolution (1D): $a=x\star w$, $a_j={\displaystyle \sum _{t=-k}^k{w}_t{x}_{j-t}}$, where $w$ is the filter, and $k$ is half of the width of the filter.
Convolution (2D): $A=X\star W$, $A_{j{j}^{\prime }}={\displaystyle \sum _{s=-k}^k{\displaystyle \sum _{t=-k}^k{W}_{s,t}{X}_{j-s,j^{\prime }-t}}}$, where $W$ is the filter, and $k$ is half of the width of the filter.
Sobel filter: $W_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]$ and $W_y=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]$.
Image gradient: ${\mathrm{\nabla }}_{x}X=W_x\star X$, ${\mathrm{\nabla }}_{y}X=W_y\star X$, with gradient magnitude $G=\sqrt{{\mathrm{\nabla }}_x^2+{\mathrm{\nabla }}_y^2}$ and gradient direction $\mathrm{\Theta }=arctan\left({\displaystyle \frac{{\mathrm{\nabla }}_y}{{\mathrm{\nabla }}_x}}\right)$.

Fully connected layer: $a=g(w^{\mathrm{\top }}x+b)$, where $a$ is the activation unit, $g$ is the activation function.
Convolution layer: $A=g(W\star X+b)$, where $A$ is the activation map.
Pooling layer: (max-pooling) $a={\displaystyle max\{x_1,...,x_m\}}$, (average-pooling) $a={\displaystyle \frac{1}{m}}{\displaystyle \sum _{j=1}^m{x}_j}$.