➩ A classifier is linear if the decision boundary is linear (line in 2D, plane in 3D, hyperplane in higher dimensions).

➩ Points above the plane (in the direction of the normal of the plane) are predicted as 1, and points below the plane are predicted as 0.

➩ A set of linear coefficients, usually estimated based on the training data, called weights, \(w_{1}, w_{2}, ..., w_{m}\), and a bias term \(b\), so that the classifier can be written in the form \(\hat{y} = 1\) if \(w_{1} x'_{1} + w_{2} x'_{2} + ... + w_{m} x'_{n} + b \geq 0\) and \(\hat{y} = 0\) if \(w_{1} x'_{1} + w_{2} x'_{2} + ... + w_{m} x'_{n} + b < 0\).

➩ Sometimes, if a probabilistic prediction is needed, \(\hat{f}\left(x'\right) = g\left(w_{1} x'_{1} + w_{2} x'_{2} + ... + w_{m} x'_{m} + b\right)\) outputs a number between 0 and 1, and the classifier can be written in the form \(\hat{y} = 1\) if \(\hat{f}\left(x'\right) \geq 0.5\) and \(\hat{y} = 0\) if \(\hat{f}\left(x'\right) < 0.5\).

➩ The function \(g\) can be any non-linear function, the resulting classifier is still linear.

➩ Linear threshold unit (LTU) perceptron: \(g\left(z\right) = 1\) if \(z > 0\) and \(g\left(z\right) = 0\) otherwise.

➩ Logistic regression: \(g\left(z\right) = \dfrac{1}{1 + e^{-z}}\), see Link.

➩ Support vector machine (SVM): \(g\left(z\right) = 1\) if \(z > 0\) and \(g\left(z\right) = 0\) otherwise, but with a different method to find the weights (that maximizes the separation between the two classes).

➩ Note: Nearest neighbors and decision trees have piecewise linear decision boundaries and is usually not considered linear classifiers.

➩ Note: Naive Bayes classifiers are not always linear under general distribution assumptions.

➩ Logistic regression is usually used for classification, not regression.

➩ The function \(g\left(z\right) = \dfrac{1}{1 + e^{-z}}\) is called the logistic activation function (or sigmoid function).

➩ How well the weights fit the training data is measured by a loss (or cost) function, which is the sum over the loss from each training item \(C\left(w\right) = C_{1}\left(w\right) + C_{2}\left(w\right) + ... + C_{n}\left(w\right)\), and for logistic regression, \(C_{i}\left(w\right) = - y_{i} \log g\left(z_{i}\right) - \left(1 - y_{i}\right) \log \left(1 - g\left(z_{i}\right)\right)\), \(z_{i} = w_{1} x_{i1} + w_{2} x_{i2} + ... + w_{m} x_{im} + b\), called the cross-entropy loss, is usually used.

➩ Given the weights and bias \(w = \left(w_{1}, w_{2}, ..., w_{m}\right), b\), the probabilistic prediction that a new item \(x' = \left(x'_{1}, x'_{2}, ..., x'_{m}\right)\) belongs to class 1 can be computed as \(\dfrac{1}{1 + e^{- \left(w_{1} x'_{1} + w_{2} x'_{2} + ... + w_{m} x'_{m} + b\right)}}\), or 1 / (1 + exp(- (x' @ w + b))), where @ is the dot product (or matrix product in general).

➩ In numpy, v @ w computes the dot product between v and w, for example, numpy.array([a, b, c]) @ numpy.array([A, B, C]) means a * A + b * B + c * C.

➩ If v and w are matrices (2D array), then v @ w computes the matrix product, which is the dot product between the rows of v and the columns of w, for example, numpy.array([[a, b, c], [d, e, f], [g, h, i]]) @ numpy.array([[A, B, C], [D, E, F], [G, H, I]]) means numpy.array([[a * A + b * D + c * G, a * B + b * E + c * H, a * C + b * F + c * I], [c * A + d * D + e * G, c * B + d * E + e * H, c * C + d * F + e * I], [g * A + h * D + i * G, g * B + h * E + i * H, g * C + h * F + i * I]).

➩ In matrix notation, \(\begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix} \begin{bmatrix} A & B & C \\ D & E & F \\ G & H & I \end{bmatrix}\) = \(\begin{bmatrix} a A + b D + c G & a B + b E + c H & a C + b F + c I \\ d A + e D + f G & d B + e E + f H & d C + e F + f I \\ g A + h D + i G & g B + h E + i H & g C + h F + i I \end{bmatrix}\).

➩ Matrix multiplications can be performed more efficiently (faster) than dot products looping over the columns and rows. More details about matrix operations in the Matrix Algebra and Linear Programming lectures.

➩ The weight \(w_{j}\) in front of feature \(j\) can be interpreted as the increase in the log-odds of the label \(y\) being 1, associated with the increase of 1 unit in \(x_{j}\), holding all other variables constant.

➩ This means, if the feature \(x'_{j}\) increases by 1, then the odds of \(y\) being 1 is increased by \(e^{w_{j}}\).

➩ The bias \(b\) is the log-odds of \(y\) being 1, when \(x'_{1} = x'_{2} = ... = x'_{m} = 0\).