Linear regression and L1/L2 regularization
Sign up for the full Machine Learning in Python course at [email protected]
Linear regression can be written in general as:
y = a0 + a1*x1 + ... + an*xn
Where y is the value of the unknown variable, and xi are the known parameters.
One option for optimizing linear regression hyperparameters is to eliminate the intercept (a0), meaning we assume that the dependent, predicted process is completely determined by the independent variables.
Another option for optimizing linear regression hyperparameters is regularization—that is, intentionally shifting the weights in the model to reduce statistical error. Naturally, to minimize the weights, the input data must be normalized.
The coefficients ai are calculated to minimize the error, i.e., Minimize the expression:
L = Σ(y - yi)^2
What if we add another term to this expression to somehow reduce the values of the coefficients ai? For example, instead of L, we could minimize the expression:
L1 = Σ(y - yi)^2 + λ1*Σ|ai|
or another option
L2 = Σ(y - yi)^2 + λ2*Σ(ai)^2
What have we done here? We've added a penalty to our expression for large values of ai. And the magnitude of this penalty is proportional to the parameter λ, which we can now use to tune our algorithm.
The first option is called L1 regularization (LASSO regression in English literature), the second option is L2 regularization, or Tikhonov regularization, or ridge regularization.
Combining these two approaches yields ElasticNet:
L = Σ(y - yi)^2 + λ1*Σ|ai| + λ2*Σ(ai)^2
Since the values of λ can be arbitrary, a "greedy" search is typically performed over a logarithmic grid of values, from 0.01 to 100, and then the resulting optimal pair of values is refined.