📗 Enter your ID (the wisc email ID without @wisc.edu) here: and click (or hit the "Enter" key) 1,2,3,4,5,6,7,8,9,10,11a3
📗 You can also load from your saved file and click .
📗 If the questions are not generated correctly, try refresh the page using the button at the top left corner.
📗 The official deadline is July 1, late submissions within one week will be accepted without penalty.
📗 The same ID should generate the same set of questions. Your answers are not saved when you close the browser. You could either copy and paste or load your program outputs into the text boxes for individual questions or print all your outputs to a single text file and load it using the button at the bottom of the page.
📗 Please do not refresh the page: your answers will not be saved.
📗 You should implement the algorithms using the mathematical formulas from the slides. You can use packages and libraries to preprocess and read the data and format the outputs. It is not recommended that you use machine learning packages or libraries, but you will not lose points for doing so.
TODO: reduce the number of images, the submission text files are too large.
1:o_axis;2:o_proj;3:o_image;4:n_axis;5:n_proj;6:n_image;7:nn_proj;8:nn_check;9:nn_image
📗 (Introduction) In this project, you will apply principal component analysis on images of hand-written digits to find the eigenvectors (eigen-hand-written-digits), and use the K-nearest neighbor algorithm to classifier new images based only on a few PCA features. By randomly perturbing the PCA features, you can reconstruct new images too.
📗 (Part 1) Read and download the training set images and labels from MNIST (this has restricted access now, please use the CSV files instead) or CSV Files (easier to read) or the same dataset in another format from other places. You can also use a smaller subset of the training set to speed up training.
📗 (Part 1) Perform PCA on the data set and compute all 784 eigenvector images. You do not have to implement eigenvalue algorithms yourself: for Java, you can use JAMA, for Python, you can use numpy or scipy (MATLAB, R, etc have packages to compute eigenvalues too). If you are interested in advanced numerical analysis, you can implement the QR algorithm too: Wikipedia. If you are unable to figure out how to do it yourself, you can use the following first \(k\) = eigenvectors:
You can find all 784 of them here: CSV file.
Plot the eigenvectors:
📗 (Part 1) Find the projection of the following \(n\) = images onto the space with the (i) original first \(k\) axes and (ii) the first \(k\) principal components as the bases, and reconstruct the images based on these \(k\) features:
You can either use the button to download a text file, or copy and paste from the text box into Excel or a csv file . Please do not change the content of the text box.
Plot the test images:
📗 (Part 2) Use 1-nearest neighbor to find an image in the training set that is the closest (but not identical) to each of the \(n\) images. Check if they indeed represent the same digit.
📗 [10 points] Enter projected lengths of the \(n\) images provided in the instructions onto the first \(k\) axes directions (\(n\) lines, each line containing \(k\) numbers, comma-separated, rounded to 4 decimal places).
Hint
📗 The projected lengths of image \(x_{i}\) (flattened into a vector of length 784) are \(\left(v^\top_{1} x_{i}, v^\top_{2} x_{i}, ..., v^\top_{10} x_{i}\right)\). The result should be \(\left(x_{i 1}, x_{i 2}, ..., x_{i 10}, 0, 0, ..., 0\right)\).
📗 [10 points] Enter the reconstructed images based on the projected lengths from the previous question (\(n\) lines, each line containing 784 numbers, comma-separated, rounded to 4 decimal places).
Hint
📗 The reconstructed image for image \(x_{i}\) is \(v^\top_{1} x_{i} v_{1} + v^\top_{2} x_{i} v_{2} + ... + v^\top_{k} x_{i} v_{k}\).
📗 [10 points] Enter the first \(k\) principal component directions as unit vectors (\(k\) lines, each line containing 784 numbers, comma-separated, rounded to 4 decimal places).
Hint
📗 Three approaches to get the principal components.
➩ Use the ones provided in the instruction (read the rows from the csv file).
➩ Construct \(X\) by putting image \(x_{i}\) in row \(i\) and subtract the mean of column \(j\) from every pixel, meaning the row \(i\) column \(j\) entry will be \(x_{ij} - \mu_{j}\), where \(\mu_{j} = \dfrac{1}{n} \displaystyle\sum_{i=1}^{n} x_{ij}\). Compute the covariance matrix product \(\Sigma = X^\top X\), and use a package to find its eigenvalues \(\lambda_{1}, \lambda_{2}, ..., \lambda_{k}\) and the corresponding eigenvectors \(u_{1}, u_{2}, ..., u_{k}\). Make sure that the eigenvalues are sorted so that \(\lambda_{1}\) is the largest, and the eigenvectors are unit vectors \(u_{k}\) satisfies \(u^\top_{k} u_{k} = 1\).
➩ (Not recommended for this course) Use QR algorithm, see Wikipedia, to compute the eigenvalues and eigenvectors: (i) start with \(\Sigma_{0} = \Sigma\), (ii) iteratively compute QR decomposition, see Wikipedia, \(\Sigma_{t} = Q_{t} R_{t}\) and update \(\Sigma_{t+1} = R_{t} Q_{t}\), (iii) repeat the process until convergence: the diagonal entries of \(\Sigma_{T}\) are the eigenvalues and the columns of \(Q_{1} Q_{2} ... Q_{T}\) are the corresponding eigenvectors. The eigenvectors computed this way might not be as accurate, but are acceptable for the purpose of this assignment.
📗 The resulting first \(k\) eigenvectors are the first \(k\) principal component directions \(u_{1}, u_{2}, ..., u_{k}\).
📗 [2 points] Enter projected lengths of the \(n\) images provided in the instructions onto the first \(k\) principal component directions (\(n\) lines, each line containing \(k\) numbers, comma-separated, rounded to 4 decimal places). These are your PCA features.
Hint
📗 The projected lengths of image \(x_{i}\) are \(\left(u^\top_{1} x_{i}, u^\top_{2} x_{i}, ..., u^\top_{k} x_{i}\right)\).
📗 Note: before doing the projection, the images can be scaled down to numbers between 0 and 1 (the original pixel intensities are values between 0 and 255). This will improve the accuracy after they are rounded to 4 decimal places.
📗 Note: please use the eigenvectors rounded to four decimal places to compute the PCA features (not the original one with higher precision since the auto-grader does not have access to those).
📗 [10 points] Enter the reconstructed images based on the PCA features from the previous question (\(n\) lines, each line containing 784 numbers, comma-separated, rounded to 4 decimal places).
Hint
📗 The reconstructed image for image \(x_{i}\) is \(u^\top_{1} x_{i} u_{1} + u^\top_{2} x_{i} u_{2} + ... + u^\top_{k} x_{i} u_{k}\).
📗 Note: please use the PCA features and eigenvectors rounded to four decimal places to compute the reconstruction (not the original ones with higher precision since the auto-grader does not have access to those).
📗 [10 points] Compute the \(k\) PCA features for all images in your training set, and enter the \(k\) PCA features of the 1 nearest neighbors for each of the \(n\) images in the training set (\(n\) lines, each line containing \(k\) numbers, comma-separated, rounded to 4 decimal places). The features should close but not identical to the ones in Question 5.
Hint
📗 Repeat Question 5 for the images in the training set.
📗 Compute the distances from each of the \(n\) PCA feature vectors from Question 5 to all the ones from the training set, and find the closest one.
📗 If the closest one is too far from the PCA feature vector from Question 5 for some image, perturb the image or the PCA features by a small amount instead.
📗 The auto-grader checks whether the features are consistent with the images in the next question, so you have to enter your answers to the next question before grading this one.
📗 [5 points] Enter the nearest neighbor images and check if they represent the same hand-written digits as the ones in the \(n\) images (\(n\) lines, each line containing 784 numbers, comma-separated, rounded to 4 decimal places).
Hint
📗 Enter the images from the training set closest corresponding to the PCA features in the previous question.
📗 [2 points] Enter the reconstructed images based on the PCA features from the previous question (\(n\) lines, each line containing 784 numbers, comma-separated, rounded to 4 decimal places).
Hint
📗 Similar to Question 6 applied to the features in Question 7.
📗 [1 points] Please confirm that you are going to submit the code on Canvas under Assignment A3, and make sure you give attribution for all blocks of code you did not write yourself (see bottom of the page for details and examples).
📗 [1 points] Please enter any comments and suggestions including possible mistakes and bugs with the questions and the auto-grading, and materials relevant to solving the question that you think are not covered well during the lectures. If you have no comments, please enter "None": do not leave it blank.
📗 Please do not modify the content in the above text field: use the "Grade" button to update.
📗 Warning: grading may take around 10 to 20 seconds. Please be patient and do not click "Grade" multiple times.
📗 You could submit multiple times (but please do not submit too often): only the latest submission will be counted.
📗 Please also save the text in the above text box to a file using the button or copy and paste it into a file yourself . You can also include the resulting file with your code on Canvas Assignment A3.
📗 You could load your answers from the text (or txt file) in the text box below using the button . The first two lines should be "##a: 3" and "##id: your id", and the format of the remaining lines should be "##1: your answer to question 1" newline "##2: your answer to question 2", etc. Please make sure that your answers are loaded correctly before submitting them.
📗 Saving and loading may take around 10 to 20 seconds. Please be patient and do not click "Load" multiple times.
📗 The sample solution in Java and Python will be posted on Piazza around the deadline. You are allowed to copy and use parts of the solution with attribution. You are allowed to use code from other people (with their permission) and from the Internet, but you must and give attribution at the beginning of the your code. You are allowed to use large language models such as GPT4 to write parts of the code for you, but you have to include the prompts you used in the code submission. For example, you can put the following comments at the beginning of your code:
% Code attribution: (TA's name)'s A3 example solution.
% Code attribution: (student name)'s A3 solution.
% Code attribution: (student name)'s answer on Piazza: (link to Piazza post).
% Code attribution: (person or account name)'s answer on Stack Overflow: (link to page).
% Code attribution: (large language model name e.g. GPT4): (include the prompts you used).
📗 You can get help on understanding the algorithm from any of the office hours; to get help with debugging, please go to the TA's office hours. For times and locations see the Home page. You are encouraged to work with other students, but if you use their code, you must give attribution at the beginning of your code.