Regularization

Regularization adds constraints or penalties to prevent overfitting.

Common Methods

Ridge Regression (L2)

$\hat{\beta }={\mathrm{argmin}}_{\beta }\Vert y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_2^2$

LASSO (L1)

$\hat{\beta }={\mathrm{argmin}}_{\beta }\Vert y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_1$

Produces sparse solutions - good for feature selection.

Elastic Net

Combines L1 and L2:

$\hat{\beta }={\mathrm{argmin}}_{\beta }\Vert y-X\beta {\Vert }_2^2+\lambda \left[\alpha \Vert \beta {\Vert }_1+(1-\alpha )\Vert \beta {\Vert }_2^2\right]$

Best for correlated features. See my blog post on Elastic Net intuition.

Connections

• Essential for High Dimensional Statistics
• Core technique in Machine Learning