9
$\begingroup$

I'm confused about the way L1 & L2 pop-up in what seem different roles in the same play:

  1. Regularization - penalty for the cost function, L1 as Lasso & L2 as Ridge
  2. Cost/Loss Function - L1 as MAE (Mean Absolute Error) and L2 as MSE (Mean Square Error)

Are [1] and [2] the same thing? or are these two completely separate practices sharing the same names? (if relevant) what are the similarities and differences between the two?

$\endgroup$
2

3 Answers 3

14
$\begingroup$

They are distinct notions.

Point #2 refers to the usual kind of loss function. The first example almost anyone who studies statistics or data analysis of any kind sees is the square loss in ordinary least squares linear regression: add up the squared residuals.

$$ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left( y_i - \hat y_i \right)^2 $$

Another viable loss function is to add up the absolute residuals.

$$ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left\vert y_i - \hat y_i \right\vert $$

Each of these can be expressed in terms of $p$-norms.

$$ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left( y_i - \hat y_i \right)^2=\vert\vert y-\hat y\vert\vert_2^2\\ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left\vert y_i - \hat y_i \right\vert = \vert\vert y-\hat y\vert\vert_1 $$

Consequently, it is reasonable to refer to these as $\ell_2$ and $\ell_1$ loss, respectively.

The penalization from the regularization in point #1 is separate. First, there is not necessarily a need to include penalization, so it might be that you just find the regression parameters that lead to predictions $\hat y_i$ giving the minimal $\ell_p$ loss, and this is exactly what ordinary least squares estimation does for $\ell_2$ loss. However, there are various reasons why the best loss value might not be desirable. Regularization is a way of sacrificing the training loss value in order to improve some other facet of performance, a major example being to sacrifice the in-sample fit of a machine learning model to quell overfitting and improve out-of-sample performance.

You can mix-and-match loss functions and regularization to your heart's content. For instance, ridge regression uses square loss with an added penalty term that involves the $\ell_2$ norm of the regression parameter vector.

$$ L_{\text{ridge}}=\vert\vert y-\hat y\vert\vert^2_2 + \lambda\vert\vert\hat\beta\vert\vert_2 $$

LASSO regression uses square loss with a penalty term that uses the $\ell_1$ norm of the parameter vector.

$$ L_{\text{LASSO}}=\vert\vert y-\hat y\vert\vert^2_2 + \lambda\vert\vert\hat\beta\vert\vert_1 $$

Elastic net uses both types of penalty.

$$ L_{\text{Elastic Net}}=\vert\vert y-\hat y\vert\vert^2_2 + \lambda_1\vert\vert\hat\beta\vert\vert_1+ \lambda_2\vert\vert\hat\beta\vert\vert_2 $$

Finally, while I do not see this approach discussed much, you could use $\ell_1$ loss with either penalty or even both.

$$ L_{\text{Other}}=\vert\vert y-\hat y\vert\vert_1 + \lambda_1\vert\vert\hat\beta\vert\vert_1+ \lambda_2\vert\vert\hat\beta\vert\vert_2 $$

(The $\lambda$ parameters control how much of a penalty there is for having large coefficients in the parameter vector. It is common to tune these using cross validation.)

Getting to other types of models, nothing stops you from using $\ell_1$ or $\ell_2$ penalization (or both) with, say, logistic regression and its associated "log loss".

$$ L_{\text{Log}}=-\overset{n}{\underset{i=1}{\sum}}\left( y_i\log(\hat y_1) + (1 - y_1)\log(1 - \hat y_1) \right)\\ L_{\text{Penalized Log}}=-\overset{n}{\underset{i=1}{\sum}}\left( y_i\log(\hat y_1) + (1 - y_1)\log(1 - \hat y_1) \right)+ \lambda_1\vert\vert\hat\beta\vert\vert_1+ \lambda_2\vert\vert\hat\beta\vert\vert_2 $$

$\endgroup$
3
  • $\begingroup$ great - very clear - small question: what is the reason that l2 as a loss function is dominant for linear regression as you noted, rather then l1 ? $\endgroup$
    – kama-shay
    Mar 20, 2023 at 11:46
  • 1
    $\begingroup$ That really warrants its own posted question (though I hope it would be a duplicate, even if I can’t think of one offhand). Briefly, there are multiple reasons, some better than others. Among the better reasons is that minimizing $\ell_2$ (\ell_2 in $\LaTeX$) coincides with maximum likelihood estimation for $iid$ Gaussian errors. This allows for many properties to be derived analytically, which was great when computing power was more limited. That leads to a less-good reason, which is tradition. $\endgroup$
    – Dave
    Mar 20, 2023 at 11:52
  • 2
    $\begingroup$ Plus of course that the two losses lead to different minimizers. Minimizing the MSE elicits conditional expectations, minimizing the MAE elicits conditional medians. (The MAE is a special case of the pinball loss, which we use to elicit conditional quantiles, and this special case elicits the 50% quantile, i.e., the median.) The two functionals coincide often, but absolutely not always, and the difference can be significant, as in modeling and forecasting intermittent demands. $\endgroup$ Mar 20, 2023 at 14:49
5
$\begingroup$

no, they refer to two different things:

  1. prior over parameters (that's your belief of how the parameters should be distributed)
  2. assumption on the "noise" of the measurements (given an observation, what's the distribution that you think describes the noise)

MAE and L1 is a Laplacian prior, MSE and L2 is a Gaussian likelihood/prior, and they are both derived using the max log-likelihood principle

$\endgroup$
5
  • 2
    $\begingroup$ There is an equivalence between regularization and putting priors on the parameters. However, you do not have to think of it that way, and I majorly object to referring to the likelihood as a prior. $\endgroup$
    – Dave
    Mar 19, 2023 at 21:03
  • $\begingroup$ @Dave sure likelihood and prior are two well distinct pieces, so that last "prior" might be a bit confusing (I'm fixing it right now) $\endgroup$
    – Alberto
    Mar 19, 2023 at 21:36
  • $\begingroup$ You also do not need to derive MSE and MAE loss as maximum likelihood estimators. There is a link between MSE and Gaussian conditional distributions (and between MAE and Laplace conditional distributions), yes, but you do not have to assume such a conditional distribution to use the loss function. // MAE also has a relationship with the likelihood, not just a prior on the parameters. // You do not have to go Bayesian to make sense of regularization. You can just apply the loss functions I show in my answer, and you just have some extremum estimator that might have nice properties. $\endgroup$
    – Dave
    Mar 19, 2023 at 21:37
  • $\begingroup$ @Dave I'm pretty sure that as long as you respect function domains (not providing a negative prediction to a BCE), you can pretty much do whatever you want, however anybody that has take a statistical class will tell you that MSE comes from that gaussian log likelihood, as that is the standard way to derive it... however, as you mentioned in your answer, nothing stops you from joining MSE and MAE, and completely ignore the statistical background behind it, and it might also work better $\endgroup$
    – Alberto
    Mar 19, 2023 at 21:41
  • 2
    $\begingroup$ You're mixing up the norm used for the loss and the norm used for the penalty, which is the exact confusion in the OP. Please read through the answer I posted. // You absolutely do not have to have a Gaussian likelihood to make sense of MSE loss. For instance, the Gauss-Markov theorem gives nice properties of the OLS linear regression estimator and does not depend on the likelihood being Gaussian. $\endgroup$
    – Dave
    Mar 19, 2023 at 21:44
4
$\begingroup$

The terms "L1" and "L2" refer to special functions called norms, which measure the length or size of a vector. You are correct in that they are used in two different contexts in statistics and machine learning, but their meanings are the same in both contexts.


In the context of regularization, the L1 and/or L2 norm restricts the magnitude of the parameter vector of a model. The difference between L1 and L2 regularization comes down to the differences between the L1 and L2 norms. See e.g. https://medium.com/analytics-vidhya/effects-of-l1-and-l2-regularization-explained-5a916ecf4f06. As pointed out in other answers, L1 regularization in a regression model corresponds to a Laplace prior on coefficients in Bayesian modeling, and L2 regularization corresponds to a Gaussian prior.

In the context of loss functions, the L1 or L2 norm measures the magnitude of the error vector of the model on a train/test/validation set. L1 loss is the Median Absolute Error (MAE), and L2 loss is the Root Mean Squared Error (RMSE). As pointed out in the comments, regression models fitted with L1 loss are models of a conditional median, while models fitted with L2 loss are models of a conditional expectation (conditional mean). The latter also happens to correspond with a Gaussian GLM maximum-likelihood model, where the conditional distribution of the data follows a Gaussian distribution centered at the regression prediction.


The L2 norm corresponds to our conventional notion of Euclidean distance, which is essentially a multi-dimensional extension of the Pythagorean theorem. You can think of Euclidean distances as the lengths of hypotenuses of right triangles drawn between points.

The L1 norm corresponds to the weirder notion of Manhattan (aka "Taxicab") distance, so named because distances resemble the distance traveled by a taxi cab following the grid layout of streets in Manhattan, New York.

It's very common in statistics and machine learning to use L2 loss (MSE) with L1 regularization, or even both L1 and L2 regularization in the same model.

L1 loss (MAE) is much less common than L2 in general, in part because the absolute value is not differentiable. However there is a "smooth" differentiable L1 loss that attempts to mimic the properties of true L1 loss, see e.g. How to interpret smooth l1 loss?.

$\endgroup$
1
  • 2
    $\begingroup$ Your first paragraph nails it. Norms are very basic principles uses in many different places. The applications can be very different concepts that only share the use of the same basic principles. $\endgroup$ Mar 20, 2023 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.