# Summary

M1, M2,
P1,
Q2,

# Lectures

Blank Slides: Part 1, Part 2,
Blank Slides (with blank pages for quiz questions): Part 1, Part 2,
Blank Slides with Quiz Questions: Part 1, Part 2,
Annotated Slides: Part 1, Part 2,


Image by Vishal Arora via medium
N/A

# Other Materials

Part 1 (Neural Network): Link
Part 2 (Backpropogation): Link
Part 3 (Multi-Layer Network): Link
Part 4 (Stochastic Gradient): Link
Part 5 (Multi-Class Classification): Link
Part 6 (Regularization): Link

Neural Network: Link
Another Neural Network Demo: Link
Neural Network Videos by Grant Sanderson: Playlist
MNIST Neural Network Visualization: Link
Neural Network Simulator: Link
Overfitting: Link
Neural Network Snake: Video
Neural Network Car: Video
Neural Network Flappy Bird: Video
Neural Network Mario: Video
MyScript: algorithm Link demo Link
Maple Calculator: Link


How to construct XOR network? Link
How derive 2-layer neural network gradient descent step? Link
How derive multi-layer neural network gradient descent induction step? Link
Comparison between L1 and L2 regularization. Link
Example (Quiz): Cross validation accuracy Link

# Keywords and Notations

Neural network classifier for two layer network with logistic activation: ${\hat{y}}_i=1_{\{a_i^{\left(2\right)}\ge 0.5\}}$
$a_{ij}^{\left(1\right)}={\displaystyle \frac{1}{1+\exp (-(\left({\displaystyle \sum _{j^{\prime }=1}^m{x}_{i{j}^{\prime }}{w}_{j^{\prime }j}^{\left(1\right)}}\right)+b_j^{\left(1\right)}))}}$, where $m$ is the number of features (or input units), $w_{j^{\prime }j}^{\left(1\right)}$ is the layer $1$ weight from input unit $j^{\prime }$ to hidden layer unit $j$, $b_j^{\left(1\right)}$ is the bias for hidden layer unit $j$, $a_{ij}^{\left(1\right)}$ is the layer $1$ activation of instance $i$ hidden unit $j$.
$a_i^{\left(2\right)}={\displaystyle \frac{1}{1+\exp (-(\left({\displaystyle \sum _{j=1}^h{a}_{ij}^{\left(1\right)}{w}_j^{\left(2\right)}}\right)+b^{\left(2\right)}))}}$, where $h$ is the number of hidden units, $w_j^{\left(2\right)}$ is the layer $2$ weight from hidden layer unit $j$, $b^{\left(2\right)}$ is the bias for the output unit, $a_i^{\left(2\right)}$ is the layer $2$ activation of instance $i$.
Stochastic gradient descent step for two layer network with squared loss and logistic activation:
$w_{j^{\prime }j}^{\left(1\right)}=w_{j^{\prime }j}^{\left(1\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})w_j^{\left(2\right)}{a}_{ij}^{\left(1\right)}(1-a_{ij}^{\left(1\right)})x_{i{j}^{\prime }}$.
$b_j^{\left(1\right)}\leftarrow b_j^{\left(1\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})w_j^{\left(2\right)}{a}_{ij}^{\left(1\right)}(1-a_{ij}^{\left(1\right)})$.
$w_j^{\left(2\right)}\leftarrow w_j^{\left(2\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})a_{ij}^{\left(1\right)}$.
$b^{\left(2\right)}\leftarrow b^{\left(2\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})$.

Softmax activation for one layer networks: $a_{ij}={\displaystyle \frac{\exp (-(w_{k^{\mathrm{\top }}}{x}_i+b_k))}{{\displaystyle \sum _{k^{\prime }=1}^K\exp (-(w_{k^{\prime }}^{\mathrm{\top }}{x}_i+b_{k^{\prime }}))}}}$, where $K$ is the number of classes (number of possible labels), $a_{ik}$ is the activation of the output unit $k$ for instance $i$, $y_{ik}$ is component $k$ of the one-hot encoding of the label for instance $i$.

L1 regularization (squared loss): ${\displaystyle \sum _{i=1}^n{(a_i-y_i)}^2+\lambda \left({\displaystyle \sum _{j=1}^m\left|w_j\right|+\left|b\right|}\right)}$, where $\lambda$ is the regularization parameter.
L2 regularization (sqaured loss): ${\displaystyle \sum _{i=1}^n{(a_i-y_i)}^2+\lambda \left({\displaystyle \sum _{j=1}^m{\left(w_j\right)}^2+b^2}\right)}$.