20
$\begingroup$

I recognize that parts of this topic have been discussed on this forum. Some examples:

What I still don't understand is why OLS regression is still the default solution to the problem of linear regression. It seems to me that in the majority of practical situations, the costs associated with errors are linear or approximately linear. If I order 2 extra parts, I incur twice the unnecessary cost as compared to if I order 1 extra part. Therefore, the optimal solution that OLS produces will not correspond to an optimal solution in reality. I understand that in general, you can minimize whatever error metric makes the most sense in the situation. My question is not about whether it is possible or a good idea to use MAE in a specific case; it is about the convention. Why is MSE minimized in nearly all simple cases instead of MAE when the real cost is typically linear?

The cases I have seen made for minimizing MSE include the following:

  1. MSE is continuously differentiable
  2. Squaring gives a greater penalty to large errors
  3. It makes sense because we assume errors to be normally distributed

However, if we can perform regression with absolute errors easily, and we are concerned primarily with prediction, I don't see how those points lead anyone to choose squared errors. If we are to choose a convention, isn't absolute error better?

This post has received many excellent answer which have all been useful to me. Of those answers, and the answers elsewhere which the mods believe answer my question, none of them exactly address the real source of my confusion except for the answer by @richard-hardy.

$\endgroup$
  • 1
    $\begingroup$ "it doesn't discuss the choice to minimize one loss over the other" - I see " in what instance would the Root Mean Squared Error be a more appropriate measure of error than the Mean Absolute Error" in that question, which seems to be precisely what you are asking about. As to why MSE is ubiquitous, well, on the one hand, it's the differentiability argument, and on the other hand, it is the only error that will be minimized by unbiased estimates/predictions, which is very often what we want. See my answer in that thread. $\endgroup$ – Stephan Kolassa Jun 5 at 20:27
  • 3
    $\begingroup$ @Aksakal: well, that is the definition of bias (except for the technical use in neural networks). And no, the fact that the (R)MSE (and nothing else) is optimized in expectation precisely by an unbiased forecast is not only true for symmetric distributions. The shoe is on the other foot: for symmetric distributions, the MAE is also minimized by an unbiased forecast (because it's minimized by the median, which is the expectation in the symmetric case), but in general, it isn't, and that's a reason to look at the MSE. $\endgroup$ – Stephan Kolassa Jun 5 at 20:39
  • 1
    $\begingroup$ I think the analytical tractability of squared loss has historically been a powerful point in its favour. $\endgroup$ – Daneel Olivaw Jun 6 at 10:46
  • 1
    $\begingroup$ Here is another related question: stats.stackexchange.com/q/369589/164061 . When we are optimizing the result of a cost function that depends on the absolute error, then it might still be useful to use estimates based on MSE. The relevant issue is not only the cost function, but also the error distribution. So... we use often MSE because that is how most error distributions are (as Gauss argued based on a few simple axioms). But for more funky distributions it makes sense to use other methods, and I believe that those are ubiquitous (only sometimes disguised as least squares like GLM). $\endgroup$ – Sextus Empiricus Jun 7 at 14:34
  • 1
    $\begingroup$ Does the duplicate question really answer the question here? There are many ways how you can discuss the relative use of MAE vs MSE. In this question the way of comparison seems to be about optimizing the result relative to some cost function. I don't see that in the other duplicate question. In none of the answers do I see an explanation relating to the sample distribution of the estimate and the idea that selecting the method which optimizes the lowest expected cost, might still be optimizing MSE even when the cost function is related to the absolute error. $\endgroup$ – Sextus Empiricus Jun 7 at 15:02
6
$\begingroup$

The first 5 answers fail to distinguish between estimation loss and prediction loss, something that is crucial in answering the question. A priori, there is no reason that the two should coincide. I will discuss both types of loss in the context of point prediction using linear regression. The discussion can be extended to models other than linear regression and tasks other than point prediction, but the essence remains the same.

Setup

Suppose you are facing a prediction problem where the model is $$ y=X\beta+\varepsilon $$ with $\varepsilon\sim D(0,\sigma)$, $D$ being some probability distribution with location $0$ and scale $\sigma$. You aim to predict $y_0$ given $x_0$, and your point prediction will be $\hat y_0$, a function of $x_0$, the data sample, the model and the penalty (the negative of reward) function defined on the prediction error. The penalty function you are facing is $L_P(y-\hat y)$. It has a minimum at zero (the value $L_P(0)$ can be set to zero without loss of generality) and is nondecreasing to both sides of zero; this is a typical characterization of a sensible prediction loss function. You can freely choose an estimation loss function $L_E(\cdot)$ and a point prediction function $y_hat_0$. What are your optimal choices for each? This will depend on the error distribution $D$ and the prediction loss function $L_P(\cdot)$.

Estimation loss

Estimation loss specifies how parameter estimates of a model are obtained from sample data. In our example of linear regression, it concern the estimation of $\beta$ and $\sigma$. You can estimate them by minimizing the sum of squared residuals (OLS) between the actual $y$ and the corresponding fitted values, sum of absolute residuals (quantile regression at the median) or another function. The choice of the estimation loss can be determined by the distribution of model errors. The most accurate estimator in some technical sense* will be achieved by the estimation loss that makes the parameter estimator the maximum likelihood (ML) estimator. If the model errors are distributed normally ($D$ is normal), this will be OLS; if they are distributed according to a Laplace distribution ($D$ is Laplace), this will be quantile regression at the mean; etc.
*To simplify, given a ML estimator, you may expect more accurate parameter estimates from your model than provided by alternative estimators.

Prediction loss

Prediction loss specifies how prediction errors are penalized. You do not choose it, it is given. (Usually, it is the client that specifies it. If the client is not capable of doing that mathematically, the analyst should strive to do that by listening carefully to the client's arguments.) If the prediction error causes the client's loss (e.g. financial loss) to grow quadratically and symmetrically about zero, you are facing square prediction loss. If the client's loss grows linearly and symmetrically about zero, you are facing absolute prediction loss. There are plenty of other possibilities for types of prediction loss you may be facing, too.

Prediction

Given the parameter estimates of the model and the values of the regressors of the point of interest, $x_0$, you should choose the point prediction $\hat y_0$ based on the prediction loss. For square loss, you will choose the estimated mean of $y_0$, as the true mean minimizes square loss on average (where the average is taken across random samples of $y_0$ subject to $x=x_0$). For absolute loss, you will choose the estimated median. For other loss function, you will choose other features of the distribution of $y_0$ that you have modelled.

Back to your question

Why do people frequently choose square error rather than absolute error, or correspondingly square loss rather than absolute loss, as estimation loss? Because normal errors ($D$ being normal) are common in applications, arguably more so than Laplace errors ($D$ being Laplace). They also make the regression estimators analytically tractable. They are not much easier to compute, however. Computational complexity of OLS (corresponding to ML estimation under normal errors) vs. quantile regression at the median (corresponding to ML estimation under Laplace errors) are not vastly different. Thus there are some sound arguments for the choice of OLS over quantile regression at the median, or square error over absolute error.

Why do people choose square error, or correspondingly square loss, as prediction loss? Perhaps for simplicity. As some of the previous answers might have mentioned, you have to choose some baseline for a textbook exposition; you cannot discuss all possible cases in detail. However, the case for prefering square loss over absolute loss as prediction loss is less convincing than in the case of estimation loss. Actual prediction loss is likely to be asymmetric (as discussed in some previous answers) and not more likely to grow quadratically than linearly with prediction error. Of course, in practice you should follow the client's specification of prediction loss. Meanwhile, in casual examples and discussions where there is no concrete client around, I do not see a strong argument for preferring square error over absolute error.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ this is absolutely on the nose and addresses precisely the points on which I was confused. Applying the method you describe in the "prediction" section for an arbitrary error distribution and cost function requires evaluating the expected value of the cost function. This requires that the product of the error distribution and cost function be integrable. Is that correct? $\endgroup$ – Ryan Volpi Jun 7 at 4:06
  • 1
    $\begingroup$ @RyanVolpi, yes, I think this is correct. Curiously, I have encountered some situations in which this is not the case; see "In model selection, what to do if expected prediction loss of all models is infinity?". $\endgroup$ – Richard Hardy Jun 7 at 10:00
  • 1
    $\begingroup$ 'Because normal errors are common in applications, arguably more so than Laplace errors' I don't think you need to caveat this with 'arguable' - Laplacian distributed variables only arise as the difference between two exponentially distributed variables, which is clearly a pretty rare situation compared to a variable which is itself the sum of many independent variables (i.e. ~Gaussian) $\endgroup$ – stuart10 Jun 8 at 8:10
  • 1
    $\begingroup$ As a corollary consider a classification problem. We typically train with a cross-entropy loss (i.e. assume the data is the result of independent Bernoulli trials), but consider metrics such a accuracy or recall when looking at validation or test data $\endgroup$ – stuart10 Jun 8 at 8:15
  • $\begingroup$ @stuart10, thanks for the comment, I have struck "arguably" out. $\endgroup$ – Richard Hardy Jun 8 at 9:01
15
$\begingroup$

TLDR; when nothing is known about actual cost of error to the user of the model, MSE is a better default option compared to MAE because, in my opinion, it is easier to manipulate analytically and is more likely to match the actual cost of error.

It's a great question. I like that you start with desire to make your loss function match actual costs. This is how it's supposed to be done ideally in my opinion. However, it is impractical to derive the cost function from actual costs every time you build a model, so we tend to gravitate to using one of the loss functions available in software. Least squares is one of the most popular functions mainly due to mathematical convenience. It is easier to deal with it analytically. Also, in some cases least squares produces unbiased point forecast, that is $E[y]-\hat y=0$, which is often considered desirable for sentimental reasons.

Having said this, I must argue that it is not obvious to me that absolute value loss is more realistic. Consider, drug overdoses - they are much costlier than underdoses in some situations: not getting high enough vs dying. Within your parts example, consider this: what if you underestimated the cost of parts to be \$1, and entered into a forward agreement to deliver one million parts one month later at \$1.1 knowing that you will have $1M one month from today. You are going to make 10% profit!

Then comes the day and parts are actually $1.2 a piece. So, you are not only going to incur loss of \$100K, but will also lack funds to deliver 1M parts. So, you are forced to default and go into bankruptcy which is very expensive. On the other hand if you overestimated the cost of parts, then you wold forego some profit but wouldn't end up in dire situation of insolvency or liquidity crisis.

This is a very common situation in business where the losses are asymmetric and highly nonlinear with rapidly escalating costs in one direction of forecast error but not the other. Hence, I'd argue that absolute loss, which is symmetric and has linear losses on forecasting error, is not realistic in most business situations. Also, although symmetric, the squared loss is at least non linear.

Yet the differences between absolute and squared loss functions don't end here. For instance, it can be shown that the optimal point forecast in absolute loss is the median while for the squared loss it is mean.

I think that the following loss function is more suitable to business forecasting in many cases where over forecasting error $e=y-\hat y$ can become very costly very quickly: $$\mathcal L(e,\hat y)=|\ln\left(1+\frac e {\hat y}\right)|$$ Here, if you are forecasting a non negative quantity $y$, then over forecasting is potentially devastating. Imagine you are bank forecasting the deposit volume, and the actual deposit volume turned out to be much lower than you hoped for. This can have severe consequences. This type of asymmetric loss function will lead to a biased optimal point forecast, i.e. $E[y]-\hat y\ne 0$, but that's exactly what you want: you want to err on the side of under forecasting in this kind of business problem.

| cite | improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ How does this apply to MSE vs MAE? This seems like cons to symmetric loss functions in general. $\endgroup$ – Dave Jun 5 at 20:12
  • 1
    $\begingroup$ +1. Very often, costs are asymmetric: if we have too much product on hand, we may be able to sell it tomorrow if we can't sell it today, but if we have too little, then we lose a sale - in such a situation it's better to over- than to underestimate. Conversely, if we are talking about strawberries, anything we don't sell today we have to throw away, so now underestimates are better than overestimates. This leads quickly to quantile regression and appropriate error measures, i.e., pinball losses. $\endgroup$ – Stephan Kolassa Jun 5 at 20:14
  • $\begingroup$ @Dave, the detailed discussion can be found in paper "Optimal Point Forecast for Certain Bank Deposit Series" see cer.columbian.gwu.edu/sites/g/files/zaxdzs2011/f/downloads/… , the PDF has an embedded paper in it. This is a pretty standard stuff though $\endgroup$ – Aksakal Jun 5 at 20:16
  • 1
    $\begingroup$ @Aksakal: I don't think I fully understand. It seems to me your loss will reward overforecasting. (Can we agree on that?) So why would we use it if overforecasting is more expensive than underforecasting? More generally, I am uncomfortable with using a loss function without knowing which functional of the future distribution it elicits - see that little paper of mine I link. I'd rather first figure out which functional I want, then choose an appropriate loss function. $\endgroup$ – Stephan Kolassa Jun 5 at 20:20
  • 2
    $\begingroup$ I think if nothing else is known then MSE is preferable to MAE. $\endgroup$ – Aksakal Jun 5 at 21:51
10
$\begingroup$

I think the reason is more sociological that statistical.

Short version: We do it this way because we always have.

Longer version: Historically, we could not do many of the things we now take for granted. Many things are computer intensive and Ronald Fisher was born before Alan Turing.

So, people did OLS regression - a lot. And people read those regressions in all sorts of substantive fields and statistics courses in those fields taught ANOVA/regression and not more modern methods.

Additionally, editors of journals learned those methods and not others, and many will reject articles with modern methods because e.g. "they won't be understood".

Many practitioners reject modern methods too; I used to be a sort of data analysis geek at a hospital. Doctors would come to ask my advice and, if it wasn't "do OLS regression" or "do logistic regression" they would reject my advice.

I got my PhD in psychometrics and many of my professors in other branches of psychology did not know any modern methods (one said: "just report the p value, that's what matters").

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I think the reason why OLS is so popular is because it started in science (Laplace etc.) where forecast error cost is symmetrical and maybe nonlinear, so it fit the most important requirements plus it's easier to manipulate analytically. If this was started in business, I bet that it wouldnt be as popular because business cost of forecast error is asymmetrical often $\endgroup$ – Aksakal Jun 5 at 20:58
  • $\begingroup$ With logistic regression you already have an example where we deviate from minimizing MSE. $\endgroup$ – Sextus Empiricus Jun 7 at 14:37
2
$\begingroup$

I think it's worth taking a step back and considering what the two losses imply.

Looking at it from a probabilistic point of view, the loss function is equivalent to the assumed log-likelihood function and thus should correspond to how we think our measurements are distributed around their unknown 'true' values.

As you say, in the case of OLS this is equivalent to assuming a Gaussian likelihood, where as an absolute error loss function is equivalent to a Laplacian likelihood. Gaussian likelihoods are far far more often a good match to real life as a consequence of the central limit theorem.

Our predictions are in general improved by making our assumed (and implicitly generative) model as close a match to reality as possible. In many (most?) cases this will improve the predictive accuracy by any sensible metric (including e.g. mean absolute error). It is far more often the case assuming a Gaussian likelihood will achieve this.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Are you claiming that a model fit by OLS will actually have a lower expected MAE on unseen data than one fit using MAE? $\endgroup$ – Ryan Volpi Jun 6 at 15:08
  • 1
    $\begingroup$ Not in absolutely all cases no, but if the underlying process producing the data is (approximately) Gaussian then assuming a squared loss in training (i.e. Gaussian likelihood) will often than not produce lower MAEs on unseen data than assuming a clearly incorrect model in training (i.e. a Laplacian likelihood). $\endgroup$ – stuart10 Jun 8 at 8:00
2
$\begingroup$

If errors are independent and follow the normal distribution (of any variance but consistent), then the sum of squared errors corresponds to their joint probability/likelihood.

$\Pi e^{-x_i^2}=e^{-\Sigma x_i^2}$

So under those conditions minimizing the sum of square errors is the same as maximizing the likelihood.


If a cost-minimizing prediction is needed (where the cost metric is different from MSE) the general/accurate approach would be to explicitly minimize the expected cost over the entire distribution of models weighted by their likelihoods (or probabilies if you have prior knowledge). This completely decouples the problem of minimizing expected cost from the problem of estimation in the presence of noise.

Suppose you are measuring a constant quantity in the presence of Gaussian noise. Even if your cost metric for future outcomes is MAE, you would rather predict with the mean (minimizing past MSE) than the median (minimizing past MAE), if indeed you know the quantity is constant and the measurement noise is Gaussian.

Example

Consider the following spread of hits produced by a gun that was mechanically fixed in place. You place a circle of a given size somewhere on the target. If the next shot lands entirely inside your circle, you win, else you lose. The cost function is of the form $f_C(x,y)=sign((x-x_C)^2+(y-y_C)^2-R^2)$.

enter image description here

If you minimize $\sum_i f_C(x_i,y_i)$, you would place the circle in the blue position, containing entirely the maximum number of past shots. But if you knew that the gun is fixed in place and the error is Gaussian, you would place the circle in the green position, centered on the data's mean/centroid (minimizing MSE), as you are optimizing future expected payoff, not average past payoff.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Is there a practical reason that maximizing likelihood would be preferred to minimizing the expectation of a realistic cost metric? $\endgroup$ – Ryan Volpi Jun 6 at 15:12
  • 1
    $\begingroup$ @RyanVolpi Consider the simplest case for example: trying to measure a constant quantity in the presence of Gaussian noise. Even if your cost metric for future outcomes is absolute error, you would rather predict with the mean (minimizing past square error) than the median (minimizing past absolute error), if indeed you know the quantity is constant and the measurement noise is Gaussian. $\endgroup$ – Museful Jun 6 at 15:50
  • 1
    $\begingroup$ @RyanVolpi In my mind minimizing cost of prediction (future) is a separate concern from suppressing measurement noise (past). The explicit way to do that would be to minimize over a distribution of weighted models. $\endgroup$ – Museful Jun 6 at 15:56
  • 1
    $\begingroup$ I never thought of it in that way. So, in the presence of gaussian noise, the mean minimizes expected MAE better than the median. Does that extend to other circumstances? For example, for a linear model with gaussian error, are the least squares estimates better than the absolute error estimates in terms of expected MAE? $\endgroup$ – Ryan Volpi Jun 6 at 17:00
  • 1
    $\begingroup$ @RyanVolpi I presume yes as long as the errors are coming (for practical purposes) from random gaussian noise and not from your model being over-constrained. $\endgroup$ – Museful Jun 6 at 17:24
1
$\begingroup$

Suppose one rolls one die (numered 1-6), and wants to compute its average deviation from the average value of 3.5. Two rolls would differ by 0.5, two by 1.5, and two by 2.5, for an average deviation of 1.5. If one takes the average of the squares of the values, one would have one deviation of 0.25, one of 2.25, and one of 6.25, for an average of 2.916 (35/12).

Now suppose instead of rolling one die, one rolls two. The average deviation would be 1.94 (35/18), and the average square of the deviation would be 5.833 (70/12).

If instead of rolling two dice, one wanted to estimate the expected deviation based upon what it was with one die, doubling the linear average single-die deviation (i.e. 1.5) would yield a value of 3, which is much larger than the actual linear average deviation of 1.94. On the other hand, doubling the average square of the deviation when using a single die (2.916) would yield precisely the average square of the deviation when using two dice.

In general, the square root of the average of the squares is a more useful number than the average of the squares itself, but if one wants to compute the square root of the average of a bunch of squares, it's easier to keep the values to be added as squares, than to take the square roots whenever reporting them and then have to square them before they can be added or averaged.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

In my opinion, it boils to that the squared error guarantees a unique solution, easier to work with and hence much more intuition. By only two main assumptions (and linearity of the error term), a quadratic loss function guarantees that the estimated coefficient is the unique minimized. Least-absolute deviations does not have this property. There is always a potential for an infinite number of solutions. Assuming that $\exists\theta_o\in\Theta$ such that $E(y|x)=m(x,\theta_o)$ and $E((m(x,\theta)-m(x,\theta_o)^2)>0$ for all $\theta\neq\theta_o$, then $\theta_o$ is the unique minimizer for non-linear least squares.

Proof: Let $y=m(x,\theta_o)+u$ and $E(u|x)=0$. Then $$E_{\theta_o}((y-m(x,\theta))^2)=E_{\theta_o}((y-m(x,\theta_o)+m(x,\theta_0)-m(x,\theta))^2)$$

$$=E_{\theta_o}(u^2)+E_{\theta_o}((m(x,\theta_o)-m(x,\theta))^2)+2E_{\theta_o}(u(m(x,\theta_o)-m(x,\theta))).$$

By the law of iterated expectations, the third term is zero. Therefore

$$E_{\theta_o}((y-m(x,\theta))^2)=u^2+E_{\theta_o}((m(x,\theta_o)-m(x,\theta))^2)$$ is uniquely minimized at $\theta_o$.

Another nice property is the total law of variance

$$Var(Y)=Var_X(E_Y(Y|X))+E_X(Var_Y(Y|X)),$$

which can be read as the variance of the dependent variable is the variance of the fitted value plus the residual's variance.

On a more technical note, the asymptotic formulas are much easier for a quadratic loss function. Importantly, the formulas don't depend on the probability density of the error term. Unfortunately, that is not true for least-absolute deviations. Therefore most practitioners end up having to assume independence of the error term (the formula has the conditional density of the error term at 0 conditioned on $x$, which is impossible to estimate($f_{u|x}(0)$)) to estimate $f_u(0)$.

And the least rigorous point is that people have an easy time understanding what a mean or expected value is, and the quadratic loss solves for the conditional expectation. Least-absolute deviations soles for the median, which is just harder to interpret. Another reason quantile regressions aren't very popular.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.