L2 or L1 penalty, maximizing likelihood

Logistic regression can be penalized with L2 or L1 to avoid overfilling and/or select variables. The idea is to maximized the likelihood. Accordingly, the total quality formula is following:

total quality = measure of fit (likelihood of the data) - measure of the magnitude of the coefficients (L2 penalty).

Why do we substract the measure of the magnitude and do not add it?

For any regression model, we try to minimize a certain loss function say $L(\beta|x)$ (e.g. sum squared residuals for OLS). Note that we can see the likelihood function as a 'negative loss function', since we try to maximize this.

In case of a penalized regression (LASSO, Ridge, etc.) we try to minimize $$L(\beta|x) + \lambda P(\beta),$$ where $P(\beta)$ is a penalizing function (e.g. $P(\beta) = ||\beta||_1 =\sum_{j=1}^p |\beta_j|$ for the LASSO). The $\lambda$ determines the size of trade off between the loss and penalizing function.

This is equivalent to maximizing (multiply by $-1$) $$-L(\beta|x) - \lambda P(\beta) = Likelihood - \lambda P(\beta)$$

Hence the subtraction of the penalty term.

EDIT: As Mr Validation pointed out in his answer, addition would favor bigger values for $\beta$. To see this, change the formula to: $$Likelihood + \lambda P(\beta),$$

with the LASSO penalizing function. If we set each $\beta_j$ very large or even $\beta_j\rightarrow\infty$, these values will produce a better maximization, however these values for $\beta$ are not realistic.

Because otherwise it would favor bigger and bigger coefficients and the estimates would blow up. You rather ask, why do we add or substract anything ;)

• This is a bit brief by our standards. Could briefly you explain, for instance, why it would favor larger coefficients? – Silverfish Aug 25 '16 at 12:39