10
$\begingroup$

This question already has an answer here:

Our statistical inference course material states the following:

The principle of mean square error can be derived from the principle of maximum likelihood (after we set a linear model where errors are normally distributed)

After that the material apparently shows this derivation over several pages of math equations with little explanation.

As I understand it, by 'principle' they mean the reason why MSE is a particularly good loss function. On another page they even explicitly state that in practice it's the most important loss function (within some context, I suppose).

Is it possible to explain without heavy math why MSE is a particularly good loss function? For example, which property makes it more important than mean absolute error (MAE)? Intuitively I would guess that MAE would be superior to MSE in most practical applications.

$\endgroup$

marked as duplicate by Kodiologist, Michael Chernick, mdewey, Peter Flom Apr 20 '17 at 11:51

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

10
$\begingroup$

To offer some alternative food for thought, here is a very simple real-world example where the MSE is not a suitable cost function, due to its symmetric nature (taken form Granger and Newbold's "Forecasting Economic Time Series" old but venerable book 2nd ed. 1986) (quote p. 125):

"A bank intends to purchase a computer to handle its current accounts. To determine the size of computer to buy, a prediction of future business is made. If the prediction is too high, the result will be that the computer will be underutilized and a cheaper machine could have been bought. If the prediction is too low, the result will be that part of the accounts will have to be handled by other means. There is no reason to suppose that the cost of errors will be symmetric in these circumstances."

When we "just" want to estimate some parameters, we have no argument to say that for example, deviating "to the right" is worse than deviating "to the left", and so symmetric cost functions are fine.

But when we want to use these statistical estimates for decision making, real-world costs enter the picture, and they very well may not be symmetrical.

See this post for a worked out example with general not-necessarily symmetric deviation costs.

Obviously the above situation is a counter-example related to all symmetric cost functions, not just MSE.

$\endgroup$
  • $\begingroup$ That is really great applied example. Thanks Alecos! Can you propose some literature on this topic? $\endgroup$ – Vivaldi Apr 20 '17 at 8:01
9
$\begingroup$

Squared Loss is differentiable, which is a very nice property. In many cases, there are close form solutions for squared loss comparing to other loss functions. Specifically, To solve the least square problem

$$ \text{minimize}~~ \|Ax-b\|^2 $$

We can set the derivative to $0$

$$2A^{T}(Ax-b)=0 $$

and solve the linear system $$A^{T}Ax=A^Tb$$.

In addition, there are many good algorithms to solve the least square problems (for example, $A^{T}Ax=A^Tb$ can be solved using QR decomposition, which uses matrix operations and has some advantages comparing to some iterative algorithms), which not for other loss functions.

I believe there are also some historical reasons, least squares are well studied for hundreds of years, and in most the text books. People use it without second thoughts on other loss functions.

$\endgroup$
  • 3
    $\begingroup$ Lots of loss functions are differentiable. I don't think that has anything to do with it. $\endgroup$ – Neil G Apr 19 '17 at 20:04
  • 4
    $\begingroup$ @NeilG: I think that hxd1011 is right in the sense that MSE is older and thus well-established, while the next-most-obvious loss function (in my opinion anyhow), absolute loss, is not differentiable. $\endgroup$ – Wayne Apr 19 '17 at 20:19
  • 1
    $\begingroup$ The mathematical reasoning used to derive linear regression requires derivatives, so if the error function is not differentiable then that same reasoning cannot be used. $\endgroup$ – James Phillips Apr 19 '17 at 20:25
  • 1
    $\begingroup$ @Wayne every exponential family has a corresponding loss function, and all of them are differentiable. And they're all, I think, equally "obvious". Logistic loss is one example. I think "older" and "well-established" is also irrelevant. $\endgroup$ – Neil G Apr 19 '17 at 20:34
  • 1
    $\begingroup$ @NeilG it's the most parsimonious differential alternative to the absolute value that I can think of. $\endgroup$ – shadowtalker Apr 19 '17 at 21:16
8
$\begingroup$

You basically answered your question in that quotation: "[it is] derived from the principle of maximum likelihood (after we set a linear model where errors are normally distributed)".

Is it reasonable to assume your errors are normally-distributed? If so, then squared loss is the most justifiable loss function because it penalizes solutions according to their log-likelihood. Penalizing according to log-likelihood is the only additive measure of loss (up to affine map) so that the loss on one data set plus the loss on another data set is the loss on the combined data set, also penalized according to log-likelihood.

I also gave a similar answer to another question (Why square the difference instead of taking the absolute value in standard deviation?).

$\endgroup$
4
$\begingroup$

This is nicely described by Christian P. Robert in The Bayesian Choice book (pp. 77-78):

Proposed by Legendre (1805) and Gauss (1810), this loss is undoubtedly the most common evaluation criterion. Founding its validity on the ambiguity of the notion of error in statistical settings (i.e., measurement error versus random variation), it also gave rise to many criticisms, commonly dealing with the fact that the squared error loss

$$ L(\theta, d) = (\theta - d)^2 \tag{2.5.1} $$

penalizes large deviations too heavily. However, convex loss functions like (2.5.1) have the incomparable advantage of avoiding the paradox of risk lovers and to exclude randomized estimators. Another usual justification for the quadratic loss is that it provides a Taylor expansion approximation to more complex symmetric losses (see Exercise 4.14 for a counterexample). In his 1810 paper, Gauss already acknowledged the arbitrariness of the quadratic loss and was defending it on grounds of simplicity. (...) In fact, the Bayes estimators associated with the quadratic loss are the posterior means. (...)

So it avoids the risk lovers paradox, since risk lovers "prefer a random gain to the expectation of this gain" (p. 59) and squared loss settles at the mean. It is connected to the notion of error and normal distribution, and it is optimization-friendly. Nonetheless, it is somehow arbitrary and certainly it is not always preferable, or ultimately the best.

$\endgroup$
  • $\begingroup$ What is the "paradox of risk lovers"? $\endgroup$ – pericynthion Apr 20 '17 at 1:39
2
$\begingroup$

Now the argument in favor of the MSE.

Consider a Loss/Cost function due to deviation, $L=L(d)$. We want it to have certain reasonable properties to do the job it is meant to do.

One such property is $L(0) = 0$.
Another is that $L(0)$ is a global minimum. But then, and if it is differentiable at $d=0$, we will have $\partial L(0)/\partial d =0$, but also $\partial^2 L(0)/\partial d^2 >0$.
A third condition is that it is everywhere increasing in $d$.

Consider now its 2nd-order Taylor expansion around zero (McLaurin):

$$L(d) \approx L(0) + \frac {\partial L(0)}{\partial d} \cdot d + \frac 12 \frac {\partial^2 L(0)}{\partial d^2}\cdot d^2 = \frac 12 \frac {\partial^2 L(0)}{\partial d^2}d^2$$

...since the first two terms are zero given the properties we want the function to have. Also, the last term is positive, and depends on the square of the deviation, so it is symmetric for negative and positive deviations.

We conclude: If as our Loss function we can use a function differentiable at $d=0$, then the deviation cost (especially for small deviations) can be acceptably modeled as a linear function of the squared deviation.

This looks like a very general and powerful argument in favor of MSE in all cases, but there are two subtle and critical points that weaken it:

1) Differentiability at $d=0$ is exactly what is lost in most cases where the real-world situation indicates that costs are asymmetric for negative and positive deviations.

2) To move from "squared error" to "expected squared error" we must consider $L(d)$ as a random variable. But then, whether one will use $E[L(d)]$ or some other "measure of concentration", becomes debatable and open to arguments, theoretical and applied.

Here is where the convenient properties of the expected value come into play, being a linear operator in theory and being estimated by sample means in applied work.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.