# Summary

M2, M3, M4, M5, M6, M7,
P1, P2, P3,
Q1, Q2, Q3, Q4, Q5, Q6,

# Lectures

N/A
N/A

# Midterm Statistics

Exam MA: Mean = 81.38%, Stdev = 14.65


Exam MB: Mean = 65.07%, Stdev = 29.1


PROB is the percentage of students who get this question correct.
RPBI is the (point biserial) correlation between getting this question correct with the total grade on the exam.
PROB < 0.25 or RPBI < 0 means this question is probably not well-made.

# Summary

W1 : M2
W2 : M3
W3 : M4
W4 : M5
W5 : M6
W6 : M7
Practice: X1 and X2 and X3 and X6

# Details

(1) The slides subtitled "Definition" and "Quiz" contain the mathematics and statistics that you are required to know for the exams.
(2) The slides subtitled "Motivation" and "Discussion" contain concepts you should be familiar with, but the specific mathematics will not be tested on the exam.
(3) The slides subtitled "Description" and "Algorithm" are mostly useful for programming homework, not exams.
(4) The slides subtitled "Admin" are not relevant to the course materials.
(1) Around a third of the questions will be exactly the same as the homework questions (different randomization of parameters), you can practice by solving these homework again with someone else's ID (auto-grading will not work if you do not enter an ID).
(2) Around a third of the questions will be similar to the past exam or quiz questions (ones that are covered during the lectures), going over the quiz questions will help, and solving the past exam questions will help.
(3) Around a third of the questions will be new, mostly from topics not covered in the homework, reading the slides will be helpful.
All questions will ask you to enter a number, vector (or list of options), or matrix. There will be no drawing or selecting objects on a canvas, and no text entry or essay questions. You will not get the hints like the ones in the homework. You can type your answers in a text file directly and submit it on Canvas. If you use the website, you can use the "calculate" button to make sure the expression you entered can be evaluated correctly when graded. You will receive 0 for incorrect answers and not-evaluate-able expressions, no partial marks, and no additional penalty for incorrect answers.

# Other Materials

No Lecture

2024 Online and In-Person Exams:
EX1: Link
CX1: Link
CX2: Link

2023 Online Exams:
M1A: Link
M2A: Link
M1B: Link
M2B: Link

2022 Online Exams:
M1A-C: Link
M2A-C: Link
MB-C: Link
MA-E: Link
MB-E: Link

2021 Online Exams:
M1A-C: Link
M1B-C: Link
M2A-C: Link
M2B-C: Link

2020 Online Exams:
M1A-C: Link
M1B-C: Link
M2A-C: Link
M2B-C: Link
M1A-E: Link
M1B-E: Link
M2A-E: Link
M2B-E: Link

2019 In-person Exams:
Midterm Version A: Link
Version A Answers: ABEDE ECDDC CCBCC CEDBB CEECD DDDBC DBBAA AAADC
Midterm Version B: Link
Version B Answers: CCABD DAECE BCADC CCEBA DDCCD DDCCA AADBC ABDAB
Sample midterm: Link


M2Q5: Link
M3Q6: Link
M4Q3: Link
M4Q6: Link
M4Q8: Link
M5Q8: Link
M6Q1: Link
M6Q4: Link
M7Q1: Link
M7Q4: Link
HMM: Link
Other Midterm Questions
Q2 (KNN Decision Boundary Question): Link
Q3 (KNN Leave-One-Out Question): Link
Q4 (CNN Count Weights Question): Link
Q5 (Naive Bayes Count CPTs Question): Link
Q7 (Please do NOT forget to submit your homework on Canvas! Question): Link

# Keywords and Notations

Training item: $(x_i,y_i)$, where $i\in \{1,2,...,n\}$ is the instance index, $x_{ij}$ is the feature $j$ of instance $i$, $j\in \{1,2,...,m\}$ is the feature index, $x_i=(x_{i1},x_{i2},....,x_{im})$ is the feature vector of instance $i$, and $y_i$ is the true label of instance $i$.
Test item: $(x^{\prime },y^{\prime })$, where $j\in \{1,2,...,m\}$ is the feature index.

LTU Classifier: ${\hat{y}}_i=1_{\{w^{\mathrm{\top }}{x}_i+b\ge 0\}}$, where $w=(w_1,w_2,...,w_m)$ is the weights, $b$ is the bias, $x_i=(x_{i1},x_{i2},...,x_{im})$ is the feature vector of instance $i$, and ${\hat{y}}_i$ is the predicted label of instance $i$.
Perceptron algorithm update step: $w=w-\alpha (a_i-y_i)x_i$, $b=b-\alpha (a_i-y_i)$, $a_i=1_{\{w^{\mathrm{\top }}{x}_i+b\ge 0\}}$, where $a_i$ is the activation value of instance $i$.

Zero-one loss minimization: $\hat{f}={\mathrm{argmin}}_{f\in \mathcal{H}}{\displaystyle \sum _{i=1}^n{1}_{\{f\left(x_i\right)\ne y_i\}}}$, where $\hat{f}$ is the optimal classifier, $\mathcal{H}$ is the hypothesis space (set of functions to choose from).
Squared loss minimization of perceptrons: $(\hat{w},\hat{b})={\mathrm{argmin}}_{w,b}{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^n{(a_i-y_i)}^2}$, $a_i=g(w^{\mathrm{\top }}{x}_i+b)$, where $\hat{w}$ is the optimal weights, $\hat{b}$ is the optimal bias, $g$ is the activation function.

Logistic regression classifier: ${\hat{y}}_i=1_{\{a_i\ge 0.5\}}$, $a_i={\displaystyle \frac{1}{1+\exp (-(w^{\mathrm{\top }}{x}_i+b))}}$.
Loss minimization problem: $(\hat{w},\hat{b})={\mathrm{argmin}}_{w,b}-{\displaystyle \sum _{i=1}^n(y_i\log \left(a_i\right)+(1-y_i)\log (1-a_i))}$, $a_i={\displaystyle \frac{1}{1+\exp (-(w^{\mathrm{\top }}{x}_i+b))}}$.
Batch gradient descrent step: $w=w-\alpha {\displaystyle \sum _{i=1}^n(a_i-y_i)x_i}$, $b=b-\alpha {\displaystyle \sum _{i=1}^n(a_i-y_i)}$, $a_i={\displaystyle \frac{1}{1+\exp (-(w^{\mathrm{\top }}{x}_i+b))}}$, where $\alpha$ is the learning rate.

Neural network classifier for two layer network with logistic activation: ${\hat{y}}_i=1_{\{a_i^{\left(2\right)}\ge 0.5\}}$
$a_{ij}^{\left(1\right)}={\displaystyle \frac{1}{1+\exp (-(\left({\displaystyle \sum _{j^{\prime }=1}^m{x}_{i{j}^{\prime }}{w}_{j^{\prime }j}^{\left(1\right)}}\right)+b_j^{\left(1\right)}))}}$, where $m$ is the number of features (or input units), $w_{j^{\prime }j}^{\left(1\right)}$ is the layer $1$ weight from input unit $j^{\prime }$ to hidden layer unit $j$, $b_j^{\left(1\right)}$ is the bias for hidden layer unit $j$, $a_{ij}^{\left(1\right)}$ is the layer $1$ activation of instance $i$ hidden unit $j$.
$a_i^{\left(2\right)}={\displaystyle \frac{1}{1+\exp (-(\left({\displaystyle \sum _{j=1}^h{a}_{ij}^{\left(1\right)}{w}_j^{\left(2\right)}}\right)+b^{\left(2\right)}))}}$, where $h$ is the number of hidden units, $w_j^{\left(2\right)}$ is the layer $2$ weight from hidden layer unit $j$, $b^{\left(2\right)}$ is the bias for the output unit, $a_i^{\left(2\right)}$ is the layer $2$ activation of instance $i$.
Stochastic gradient descent step for two layer network with squared loss and logistic activation:
$w_{j^{\prime }j}^{\left(1\right)}=w_{j^{\prime }j}^{\left(1\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})w_j^{\left(2\right)}{a}_{ij}^{\left(1\right)}(1-a_{ij}^{\left(1\right)})x_{i{j}^{\prime }}$.
$b_j^{\left(1\right)}\leftarrow b_j^{\left(1\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})w_j^{\left(2\right)}{a}_{ij}^{\left(1\right)}(1-a_{ij}^{\left(1\right)})$.
$w_j^{\left(2\right)}\leftarrow w_j^{\left(2\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})a_{ij}^{\left(1\right)}$.
$b^{\left(2\right)}\leftarrow b^{\left(2\right)}-\alpha (a_i^{\left(2\right)}-y_i)a_i^{\left(2\right)}(1-a_i^{\left(2\right)})$.

Softmax activation for one layer networks: $a_{ij}={\displaystyle \frac{\exp (-(w_{k^{\mathrm{\top }}}{x}_i+b_k))}{{\displaystyle \sum _{k^{\prime }=1}^K\exp (-(w_{k^{\prime }}^{\mathrm{\top }}{x}_i+b_{k^{\prime }}))}}}$, where $K$ is the number of classes (number of possible labels), $a_{ik}$ is the activation of the output unit $k$ for instance $i$, $y_{ik}$ is component $k$ of the one-hot encoding of the label for instance $i$.

L1 regularization (squared loss): ${\displaystyle \sum _{i=1}^n{(a_i-y_i)}^2+\lambda \left({\displaystyle \sum _{j=1}^m\left|w_j\right|+\left|b\right|}\right)}$, where $\lambda$ is the regularization parameter.
L2 regularization (sqaured loss): ${\displaystyle \sum _{i=1}^n{(a_i-y_i)}^2+\lambda \left({\displaystyle \sum _{j=1}^m{\left(w_j\right)}^2+b^2}\right)}$.

SVM classifier: ${\hat{y}}_i=1_{\{w^{\mathrm{\top }}{x}_i+b\ge 0\}}$.
Hard margin, original max-margin formulation: ${\displaystyle \underset{w}{max}{\displaystyle \frac{2}{\sqrt{w^{\mathrm{\top }}w}}}}$ such that $w^{\mathrm{\top }}{x}_i+b\le -1$ if $y_i=0$ and $w^{\mathrm{\top }}{x}_i+b\ge 1$ if $y_i=1$.
Hard margin, simplified formulation: ${\displaystyle \underset{w}{min}{\displaystyle \frac{1}{2}}{w}^{\mathrm{\top }}w}$ such that $(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\ge 1$.
Soft margin, original max-margin formulation: ${\displaystyle \underset{w}{min}{\displaystyle \frac{1}{2}}{w}^{\mathrm{\top }}w+{\displaystyle \frac{1}{\lambda }}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{\xi }_i}}$ such that $(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\ge 1-\xi ,\xi \ge 0$, where ${\xi }_i$ is the slack variable for instance $i$, $\lambda$ is the regularization parameter.
Soft margin, simplified formulation: ${\displaystyle \underset{w}{min}{\displaystyle \frac{\lambda }{2}}{w}^{\mathrm{\top }}w+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{\displaystyle max\{0,1-(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\}}}}$
Subgradient descent formula: $w=(1-\lambda )w-\alpha (2y_i-1)1_{\{(2y_i-1)(w^{\mathrm{\top }}{x}_i+b)\ge 1\}}{x}_i$.

Kernel SVM classifier: ${\hat{y}}_i=1_{\{w^{\mathrm{\top }}\phi \left(x_i\right)+b\ge 0\}}$, where $\phi$ is the feature map.
Kernal Gram matrix: $K_{i{i}^{\prime }}=\phi {\left(x_i\right)}^{\mathrm{\top }}\phi \left(x_{i^{\prime }}\right)$.
Quadratic Kernel: $K_{i{i}^{\prime }}={(x_{i^{\mathrm{\top }}}{x}_{i^{\prime }}+1)}^2$ has feature representation $\phi \left(x_i\right)=(x_{i1}^2,x_{i2}^2,\sqrt{2}{x}_{i1}{x}_{i2},\sqrt{2}{x}_{i1},\sqrt{2}{x}_{i2},1)$.
Gaussian RBF Kernel: $K_{i{i}^{\prime }}=\exp (-{\displaystyle \frac{1}{2{\sigma }^2}}{(x_i-x_{i^{\prime }})}^{\mathrm{\top }}(x_i-x_{i^{\prime }}))$ has infinite-dimensional feature representation, where ${\sigma }^2$ is the variance parameter.

Entropy: $H\left(Y\right)=-{\displaystyle \sum _{y=1}^K{p}_y{\log }_2\left(p_y\right)}$, where $K$ is the number of classes (number of possible labels), $p_y$ is the fraction of data points with label $y$.
Conditional entropy: $H\left(Y|X\right)=-{\displaystyle \sum _{x=1}^{K_X}{p}_x{\displaystyle \sum _{y=1}^K{p}_{y|x}{\log }_2\left(p_{y|x}\right)}}$, where $K_X$ is the number of possible values of feature, $p_x$ is the fraction of data points with feature $x$, $p_{y|x}$ is the fraction of data points with label $y$ among the ones with feature $x$.
Information gain, for feature $j$: $I\left(Y|X_j\right)=H\left(Y\right)-H\left(Y|X_j\right)$.

Decision stump classifier: ${\hat{y}}_i=1_{\{x_{ij}\ge t_j\}}$, where $t_j$ is the threshold for feature $j$.
Feature selection: $j^{\star }={\mathrm{argmax}}_{j}I\left(Y|X_j\right)$.

Distance: (Euclidean) $\rho (x,x^{\prime })={\Vert x-x^{\prime }\Vert }_2=\sqrt{{\displaystyle \sum _{j=1}^m{(x_j-x_j^{\prime })}^2}}$, (Manhattan) $\rho (x,x^{\prime })={\Vert x-x^{\prime }\Vert }_1={\displaystyle \sum _{j=1}^m|x_j-x_j^{\prime }|}$, where $x,x^{\prime }$ are two instances.
K-Nearest Neighbor classifier: ${\hat{y}}_i$ = mode $\{y_{\left(1\right)},y_{\left(2\right)},...,y_{\left(k\right)}\}$, where mode is the majority label and $y_{\left(t\right)}$ is the label of the $t$-th closest instance to instance $i$ from the training set.

Unigram model: $\mathbb{P}\{z_1,z_2,...,z_d\}={\displaystyle \prod _{t=1}^d\mathbb{P}\left\{z_t\right\}}$ where $z_t$ is the $t$-th token in a training item, and $d$ is the total number of tokens in the item.
Maximum likelihood estimator (unigram): $\hat{\mathbb{P}}\left\{z_t\right\}={\displaystyle \frac{c_{z_t}}{{\displaystyle \sum _{z=1}^m{c}_z}}}$, where $c_z$ is the number of time the token $z$ appears in the training set and $m$ is the vocabulary size (number of unique tokens).
Maximum likelihood estimator (unigram, with Laplace smoothing): $\hat{\mathbb{P}}\left\{z_t\right\}={\displaystyle \frac{c_{z_t}+1}{\left({\displaystyle \sum _{z=1}^m{c}_z}\right)+m}}$.
Bigram model: $\mathbb{P}\{z_1,z_2,...,z_d\}=\mathbb{P}\left\{z_1\right\}{\displaystyle \prod _{t=2}^d\mathbb{P}\left\{z_t|z_{t-1}\right\}}$.
Maximum likelihood estimator (bigram): $\hat{\mathbb{P}}\left\{z_t|z_{t-1}\right\}={\displaystyle \frac{c_{z_{t-1},z_t}}{c_{z_{t-1}}}}$.
Maximum likelihood estimator (bigram, with Laplace smoothing): $\hat{\mathbb{P}}\left\{z_t|z_{t-1}\right\}={\displaystyle \frac{c_{z_{t-1},z_t}+1}{c_{z_{t-1}}+m}}$.

Conditional probability: $\mathbb{P}\{Y=y|X=x\}={\displaystyle \frac{\mathbb{P}\{Y=y,X=x\}}{\mathbb{P}\{X=x\}}}$.
Joint probability: $\mathbb{P}\{X=x\}={\displaystyle \sum _{y\in Y}\mathbb{P}\{X=x,Y=y\}}$.
Bayes rule: $\mathbb{P}\{Y=y|X=x\}={\displaystyle \frac{\mathbb{P}\{X=x|Y=y\}\mathbb{P}\{Y=y\}}{{\displaystyle \sum _{y^{\prime }\in Y}\mathbb{P}\{X=x|Y=y^{\prime }\}\mathbb{P}\{Y=y^{\prime }\}}}}$.
Law of total probability: $\mathbb{P}\{X=x\}={\displaystyle \sum _{y^{\prime }\in Y}\mathbb{P}\{X=x|Y=y^{\prime }\}\mathbb{P}\{Y=y^{\prime }\}}$.
Independence: $X,Y$ are independent if $\mathbb{P}\{X=x,Y=y\}=\mathbb{P}\{X=x\}\mathbb{P}\{Y=y\}$ for every $x,y$.
Conditional independence: $X,Y$ are conditionally independent conditioned on $Z$ if $\mathbb{P}\{X=x,Y=y|Z=z\}=\mathbb{P}\{X=x|Z=z\}\mathbb{P}\{Y=y|Z=z\}$ for every $x,y,z$.

Conditional Probability Table estimation: $\hat{\mathbb{P}}\left\{x_j|p\left(X_j\right)\right\}={\displaystyle \frac{c_{x_j,p\left(X_j\right)}}{c_{p\left(X_j\right)}}}$, where $p\left(X_j\right)$ is the list of parents of $X_j$ in the network.
Conditional Probability Table estimation (with Laplace smoothing): $\hat{\mathbb{P}}\left\{x_j|p\left(X_j\right)\right\}={\displaystyle \frac{c_{x_j,p\left(X_j\right)}+1}{c_{p\left(X_j\right)}+\left|X_j\right|}}$, where $\left|X_j\right|$ is the number of possible values of $X_j$.
Bayesian network inference: $\mathbb{P}\{x_1,x_2,...,x_m\}={\displaystyle \prod _{j=1}^m\mathbb{P}\left\{x_j|p\left(X_j\right)\right\}}$.
Naive Bayes estimation: .
Naive Bayes classifier: ${\hat{y}}_i={\mathrm{argmax}}_y\mathbb{P}\{Y=y|X=X_i\}$.

Convolution (1D): $a=x\star w$, $a_j={\displaystyle \sum _{t=-k}^k{w}_t{x}_{j-t}}$, where $w$ is the filter, and $k$ is half of the width of the filter.
Convolution (2D): $A=X\star W$, $A_{j{j}^{\prime }}={\displaystyle \sum _{s=-k}^k{\displaystyle \sum _{t=-k}^k{W}_{s,t}{X}_{j-s,j^{\prime }-t}}}$, where $W$ is the filter, and $k$ is half of the width of the filter.
Sobel filter: $W_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]$ and $W_y=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]$.
Image gradient: ${\mathrm{\nabla }}_{x}X=W_x\star X$, ${\mathrm{\nabla }}_{y}X=W_y\star X$, with gradient magnitude $G=\sqrt{{\mathrm{\nabla }}_x^2+{\mathrm{\nabla }}_y^2}$ and gradient direction $\mathrm{\Theta }=arctan\left({\displaystyle \frac{{\mathrm{\nabla }}_y}{{\mathrm{\nabla }}_x}}\right)$.

Fully connected layer: $a=g(w^{\mathrm{\top }}x+b)$, where $a$ is the activation unit, $g$ is the activation function.
Convolution layer: $A=g(W\star X+b)$, where $A$ is the activation map.
Pooling layer: (max-pooling) $a={\displaystyle max\{x_1,...,x_m\}}$, (average-pooling) $a={\displaystyle \frac{1}{m}}{\displaystyle \sum _{j=1}^m{x}_j}$.