📗 [4 points] For a reinforcement learning problem, there are states and actions. Each state can be represented by features. Deep Q Networks are used to represent the Q function: there is only one hidden layer with \(n\) hidden units, the input units represent the features of the states, and output units represent the Q value for each action. What is the smallest value of \(n\) so that the number of weights plus biases of the network is strictly larger than the number of Q values in the original Q table?

📗 Answer: Calculate

📗 [4 points] (NEW) Consider the problem of driving a car on a by grid, where the car can move up, down, left, right, or stay in the same position. The states are encoded by two features, the x and y coordinates of the position of the car, a policy neural network is trained using the policy gradient algorithm REINFORCE. Suppose the policy network has two ReLU hidden layers with and hidden units and a softmax output layer. How many weights (excluding biases) are updated in each iteration of REINFORCE?

📗 Answer: Calculate

📗 [4 points] (NEW) In an attention unit in a not-so-large language model, given a sentence "", the value units \(V = a^{\left(v\right)} = w^{\left(v\right)} \cdot x\) = , the key units \(K = a^{\left(k\right)} = w^{\left(k\right)} \cdot x\) = , and the query unit for the word "" \(q = a^{\left(q\right)} = w^{\left(q\right)} x\) = . What is the vector of attention weights (also called soft weights) for the word ""? Use the scaled dot product attention \(\text{softmax} \left(\dfrac{q K^\top}{\sqrt{d}}\right)\).

📗 Answer (comma separated vector, with values sum up to 1): Calculate

📗 [4 points] (NEW) Suppose an auto-encoder neural network (fully connected) with one hidden layer with hidden units is used to perform non-linear dimensionality reduction for the bag-of-words features with a vocabulary size of so that the documents can be visualized as 3D points. When training the network (both encoder and decoder), how many weights (excluding biases) are updated?

📗 Answer: Calculate