# Lecture

📗 The lecture is in person, but you can join Zoom: 8:50-9:40 or 11:00-11:50. Zoom recordings can be viewed on Canvas -> Zoom -> Cloud Recordings. They will be moved to Kaltura over the weekends.
📗 The in-class (participation) quizzes should be submitted on TopHat (Code:741565), but you can submit your answers through Form at the end of the lectures too.
📗 The Python notebooks used during the lectures can also be found on: GitHub. They will be updated weekly.

# Lecture Notes

TopHat Game Again

➭ There will be 20 questions on the exam, 10 of them from past exams and quizzes, and 10 of them new questions (see Link for details). I will post \(n\) more questions next Monday that are identical or similar to \(n\) of the new questions on exam.
➭ A: \(n = 0\)
➭ B: \(n = 1\) if more than 50 percent of you choose B.
➭ C: \(n = 2\) if more than 75 percent of you choose C.
➭ D: \(n = 3\) if more than 95 percent of you choose D.
➭ E: \(n = 0\)

📗 Dimensionality Reduction
➭ Dimensionality reduction finds a low dimensional representation of a high dimensional point (item) in a way that points that close to each other in the original space are close to each other in the low dimensional space.
➭ Text and image data have a large number of features (dimensions), dimensionality reduction techniques can be used for visualization and efficient storage of these data types: Link.



📗 Principal Component Analysis
➭ Principal component analysis finds the orthogonal directions that capture the highest variation, called principal components, and project the feature vectors onto the subspace generated by the principal components: Link.

📗 Projection
➭ A unit vector \(u\) in the direction of a vector \(v\) is \(u = \dfrac{v}{\left\|v\right\|}\), where \(\left\|v\right\| = \sqrt{v_{1}^{2} + v_{2}^{2} + ... v_{m}^{2}}\). Every unit vector satisfy \(u^\top u = 1\).
➭ The projection of a feature vector \(x\) onto the direction of a unit vector \(u\) is \(u^\top x\).
➭ Geometrically, the projection of \(x\) onto \(u\) finds the vector in the direction of \(u\) that is the closest to \(x\).

📗 Projected Variances
➭ Projected variance along a direction \(u\) is can be computed by \(u^\top \Sigma u\), so the PCA problem is given by the constrained optimization problem \(\displaystyle\max_{u} u^\top \Sigma u\) subject to \(u^\top u = 1\).
➭ A closed-form solution is \(\Sigma u = \lambda u\), which means \(\lambda\) is an eigenvalue of \(\Sigma\) and \(u\) is an eigenvector of \(\Sigma\).
➭ A faster way to find the eigenvalue-eigenvector pair is through singular value decomposition (SVD), and it is used in sklearn.decomposition.PCA: Doc.

US States Economics Data Again, Again

➭ Reduce the dimension of the US states economic data set:
➭ Code for PCA with one dimension: Notebook.
➭ Code for PCA with two dimensions: Notebook.
➭ Code for PCA with three dimensions: Notebook.
➭ Start with 15 features and find the number of PCA dimensions that can explain 99 percent of the variance.
➭ Code for PCA with 99 percent threshold: Notebook.

📗 Reconstruction
➭ If there are \(m\) original features, and \(m\) principal components are computed, then the original item can be perfectly reconstructed, if only \(k < m\) principal components are used, then the original item can be approximated.

EigenFace Example

➭ EigenFaces are principal components of face images.
➭ Since principal components have the same dimensions as the original items, EigenFaces are images with the same dimensions as the original face images.
➭ Eigenfaces example:

Link
➭ Reconstruction examples:

K	1	20	40	50	80
Face 1
Face 2

Images by wellecks

📗 Number of Reduced Dimensions
➭ Similar to other clustering methods, the number of dimensions \(K\) are usually chosen based on application requirements (for example, 2 or 3 for visualization).
➭ sklearn.decomposition.PCA(k) can take in k values between 0 and 1 too, and in that case, k represents the target amount of variance that needs to be explained.

📗 Non-linear PCA
➭ If the kernel trick is used to create new features for PCA, then PCA features are non-linear in the original features, and it's called kernel PCA. sklearn.decomposition.KernelPCA computes the kernel PCA: Doc.

📗 Auto-Encoder
➭ If a neural network is trained with \(y = x\), then the hidden layer unit values can be viewed as "encoding" of \(x\).
➭ Auto-encoder is a non-linear dimensionality reduction method if the neural network has nonlinear activations.

Additional Examples

➭ Suppose the explained_variance_ratio of a PCA model is [0.4, 0.3, 0.2, 0.1]. How many components (at least) are needed to explain 80 percent (or more) of the variance of the original data?
➭ 1 component can explain 40 percent.
➭ 2 components can explain 40 + 30 = 70 percent.
➭ 3 components can explain 40 + 30 + 20 = 90 percent, which is more than 80.
➭ At least 3 components are need.

📗 Notes and code adapted from the course taught by Yiyin Shen Link and Tyler Caraza-Harter Link