➩ A linear program is an optimization problem in which the objective is linear and the constraints are linear.

➩ The set of feasible solutions are the ones that satisfy the constraints, but may or may not be optimal.

➩ The feasible solutions of linear programs form a convex polytope (line segment in 1D, convex polygon in 2D).

➩ The standard form of a linear program is: \(\displaystyle\max_{x} c^\top x\) subject to \(A x \leq b\) and \(x \geq 0\).

➩ For example, if there are two variables and two constraints, then the standard form is: \(\displaystyle\max_{x_{1}, x_{2}} c_{1} x_{1} + c_{2} x_{2}\) subject to \(A_{11} x_{1} + A_{12} x_{2} \leq b_{1}\), \(A_{21} x_{1} + A_{22} x_{2} \leq b_{2}\) and \(x_{1}, x_{2} \geq 0\), which in matrix form, it is \(\displaystyle\max_{\begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix}} \begin{bmatrix} c_{1} & c_{2} \end{bmatrix} \begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix}\) subject to \(\begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix} \begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix} \leq \begin{bmatrix} b_{1} \\ b_{2} \end{bmatrix}\) and \(\begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix} \geq \begin{bmatrix} 0 \\ 0 \end{bmatrix}\).

➩ The dual form of a linear program in standard form is: \(\displaystyle\min b^\top y\) subject to \(A^\top y \geq c\) and \(y \geq 0\).

➩ For example, the dual form of the two variable is, \(\displaystyle\min_{y_{1}, y_{2}} b_{1} y_{1} + b_{2} y_{2}\) subject to \(A_{11} y_{1} + A_{21} y_{2} \geq c_{1}\), \(A_{12} y_{1} + A_{22} y_{2} \geq c_{2}\) and \(y_{1}, y_{2} \geq 0\), which in matrix form, it is \(\displaystyle\max_{\begin{bmatrix} y_{1} \\ y_{2} \end{bmatrix}} \begin{bmatrix} b_{1} & b_{2} \end{bmatrix} \begin{bmatrix} y_{1} \\ y_{2} \end{bmatrix}\) subject to \(\begin{bmatrix} A_{11} & A_{21} \\ A_{12} & A_{22} \end{bmatrix} \begin{bmatrix} y_{1} \\ y_{2} \end{bmatrix} \geq \begin{bmatrix} c_{1} \\ c_{2} \end{bmatrix}\) and \(\begin{bmatrix} y_{1} \\ y_{2} \end{bmatrix} \geq \begin{bmatrix} 0 \\ 0 \end{bmatrix}\).

➩ Duality theorem says that the primal and dual solutions lead to the same objective value.

➩ scipy.optimize.linprog provides two methods of solving a linear program: Doc

➩ The problem of regression by minimizing the sum of absolute value of the error (instead of the square), that is, \(C\left(w\right) = \displaystyle\min_{w} \displaystyle\sum_{i=1}^{n} \left| y_{i} - \left(w_{1} x_{i1} + w_{2} x_{i2} + ... + w_{m} x_{im} + b\right) \right|\), can be written as a linear program.

➩ The can be done by noting \(\left| a \right| = \displaystyle\max\left\{a, -a\right\}\), so the problem can be written as, \(\displaystyle\min_{a, w, b} \displaystyle\sum_{i=1}^{n} a_{i}\) subject to \(a_{i} \geq y_{i} - \left(w_{1} x_{i1} + w_{2} x_{i2} + ... + w_{m} x_{im} + b\right)\) and \(a_{i} \geq -\left(y_{i} - \left(w_{1} x_{i1} + w_{2} x_{i2} + ... + w_{m} x_{im} + b\right)\right)\).

➩ The resulting regression problem is also called Least Absolute Deviations (LAD): Link.

➩ The problem of finding the optimal weights for a support vector machine can be written as minimizing the hinge loss \(C\left(w\right) = \displaystyle\sum_{i=1}^{n} \displaystyle\max\left\{1 - \left(2 y_{i} - 1\right) \cdot \left(w_{1} x_{i1} + w_{2} x_{i2} + ... + w_{m} x_{im} + b\right), 0\right\}\), which can be converted into a linear program.

➩ Similar to the regression problem, the problem can be written as, \(\displaystyle\min_{a, w, b} \displaystyle\sum_{i=1}^{n} a_{i}\) subject to \(a_{i} \geq 1 - \left(w_{1} x_{i1} + w_{2} x_{i2} + ... + w_{m} x_{im} + b\right)\) if \(y_{i} = 1\), and \(a_{i} \geq 1 + \left(w_{1} x_{i1} + w_{2} x_{i2} + ... + w_{m} x_{im} + b\right)\) if \(y_{i} = 0\), and \(a_{i} \geq 0\).

➩ When kernel trick is used, the dual of the linear program with the new features, possibly infinite number of them, is finite and can be solved instead.