# L2 or L1 penalty, maximizing likelihood

Logistic regression can be penalized with L2 or L1 to avoid overfilling and/or select variables. The idea is to maximized the likelihood. Accordingly, the total quality formula is following:

total quality = measure of fit (likelihood of the data) - measure of the magnitude of the coefficients (L2 penalty).


Why do we substract the measure of the magnitude and do not add it?

## 2 Answers

For any regression model, we try to minimize a certain loss function say $L(\beta|x)$ (e.g. sum squared residuals for OLS). Note that we can see the likelihood function as a 'negative loss function', since we try to maximize this.

In case of a penalized regression (LASSO, Ridge, etc.) we try to minimize $$L(\beta|x) + \lambda P(\beta),$$ where $P(\beta)$ is a penalizing function (e.g. $P(\beta) = ||\beta||_1 =\sum_{j=1}^p |\beta_j|$ for the LASSO). The $\lambda$ determines the size of trade off between the loss and penalizing function.

This is equivalent to maximizing (multiply by $-1$) $$-L(\beta|x) - \lambda P(\beta) = Likelihood - \lambda P(\beta)$$

Hence the subtraction of the penalty term.

EDIT: As Mr Validation pointed out in his answer, addition would favor bigger values for $\beta$. To see this, change the formula to: $$Likelihood + \lambda P(\beta),$$

with the LASSO penalizing function. If we set each $\beta_j$ very large or even $\beta_j\rightarrow\infty$, these values will produce a better maximization, however these values for $\beta$ are not realistic.

Because otherwise it would favor bigger and bigger coefficients and the estimates would blow up. You rather ask, why do we add or substract anything ;)

• This is a bit brief by our standards. Could briefly you explain, for instance, why it would favor larger coefficients? – Silverfish Aug 25 '16 at 12:39