What makes mean square error so good? [duplicate]

Question

Our statistical inference course material states the following:

The principle of mean square error can be derived from the principle of maximum likelihood (after we set a linear model where errors are normally distributed)

After that the material apparently shows this derivation over several pages of math equations with little explanation.

As I understand it, by 'principle' they mean the reason why MSE is a particularly good loss function. On another page they even explicitly state that in practice it's the most important loss function (within some context, I suppose).

Is it possible to explain without heavy math why MSE is a particularly good loss function? For example, which property makes it more important than mean absolute error (MAE)? Intuitively I would guess that MAE would be superior to MSE in most practical applications.

Answer 1

To offer some alternative food for thought, here is a very simple real-world example where the MSE is not a suitable cost function, due to its symmetric nature (taken form Granger and Newbold's "Forecasting Economic Time Series" old but venerable book 2nd ed. 1986) (quote p. 125):

"A bank intends to purchase a computer to handle its current accounts. To determine the size of computer to buy, a prediction of future business is made. If the prediction is too high, the result will be that the computer will be underutilized and a cheaper machine could have been bought. If the prediction is too low, the result will be that part of the accounts will have to be handled by other means. There is no reason to suppose that the cost of errors will be symmetric in these circumstances."

When we "just" want to estimate some parameters, we have no argument to say that for example, deviating "to the right" is worse than deviating "to the left", and so symmetric cost functions are fine.

But when we want to use these statistical estimates for decision making, real-world costs enter the picture, and they very well may not be symmetrical.

See this post for a worked out example with general not-necessarily symmetric deviation costs.

Obviously the above situation is a counter-example related to all symmetric cost functions, not just MSE.

Answer 2

Squared Loss is differentiable, which is a very nice property. In many cases, there are close form solutions for squared loss comparing to other loss functions. Specifically, To solve the least square problem

$$ \text{minimize}~~ \|Ax-b\|^2 $$

We can set the derivative to $0$

$$2A^{T}(Ax-b)=0 $$

and solve the linear system $$A^{T}Ax=A^Tb$$.

In addition, there are many good algorithms to solve the least square problems (for example, $A^{T}Ax=A^Tb$ can be solved using QR decomposition, which uses matrix operations and has some advantages comparing to some iterative algorithms), which not for other loss functions.

I believe there are also some historical reasons, least squares are well studied for hundreds of years, and in most the text books. People use it without second thoughts on other loss functions.

Answer 3

You basically answered your question in that quotation: "[it is] derived from the principle of maximum likelihood (after we set a linear model where errors are normally distributed)".

Is it reasonable to assume your errors are normally-distributed? If so, then squared loss is the most justifiable loss function because it penalizes solutions according to their log-likelihood. Penalizing according to log-likelihood is the only additive measure of loss (up to affine map) so that the loss on one data set plus the loss on another data set is the loss on the combined data set, also penalized according to log-likelihood.

I also gave a similar answer to another question (Why square the difference instead of taking the absolute value in standard deviation?).

Answer 4

This is nicely described by Christian P. Robert in The Bayesian Choice book (pp. 77-78):

Proposed by Legendre (1805) and Gauss (1810), this loss is undoubtedly the most common evaluation criterion. Founding its validity on the ambiguity of the notion of error in statistical settings (i.e., measurement error versus random variation), it also gave rise to many criticisms, commonly dealing with the fact that the squared error loss

$$ L(\theta, d) = (\theta - d)^2 \tag{2.5.1} $$

penalizes large deviations too heavily. However, convex loss functions like (2.5.1) have the incomparable advantage of avoiding the paradox of risk lovers and to exclude randomized estimators. Another usual justification for the quadratic loss is that it provides a Taylor expansion approximation to more complex symmetric losses (see Exercise 4.14 for a counterexample). In his 1810 paper, Gauss already acknowledged the arbitrariness of the quadratic loss and was defending it on grounds of simplicity. (...) In fact, the Bayes estimators associated with the quadratic loss are the posterior means. (...)

So it avoids the risk lovers paradox, since risk lovers "prefer a random gain to the expectation of this gain" (p. 59) and squared loss settles at the mean. It is connected to the notion of error and normal distribution, and it is optimization-friendly. Nonetheless, it is somehow arbitrary and certainly it is not always preferable, or ultimately the best.

Answer 5

Now the argument in favor of the MSE.

Consider a Loss/Cost function due to deviation, $L=L(d)$. We want it to have certain reasonable properties to do the job it is meant to do.

One such property is $L(0) = 0$.
Another is that $L(0)$ is a global minimum. But then, and if it is differentiable at $d=0$, we will have $\partial L(0)/\partial d =0$, but also $\partial^2 L(0)/\partial d^2 >0$.
A third condition is that it is everywhere increasing in $d$.

Consider now its 2nd-order Taylor expansion around zero (McLaurin):

$$L(d) \approx L(0) + \frac {\partial L(0)}{\partial d} \cdot d + \frac 12 \frac {\partial^2 L(0)}{\partial d^2}\cdot d^2 = \frac 12 \frac {\partial^2 L(0)}{\partial d^2}d^2$$

...since the first two terms are zero given the properties we want the function to have. Also, the last term is positive, and depends on the square of the deviation, so it is symmetric for negative and positive deviations.

We conclude: If as our Loss function we can use a function differentiable at $d=0$, then the deviation cost (especially for small deviations) can be acceptably modeled as a linear function of the squared deviation.

This looks like a very general and powerful argument in favor of MSE in all cases, but there are two subtle and critical points that weaken it:

1) Differentiability at $d=0$ is exactly what is lost in most cases where the real-world situation indicates that costs are asymmetric for negative and positive deviations.

2) To move from "squared error" to "expected squared error" we must consider $L(d)$ as a random variable. But then, whether one will use $E[L(d)]$ or some other "measure of concentration", becomes debatable and open to arguments, theoretical and applied.

Here is where the convenient properties of the expected value come into play, being a linear operator in theory and being estimated by sample means in applied work.