# Bias towards natural numbers in the case of least squares

Why do we seek to minimize x^2 instead of minimizing |x|^1.95 or |x|^2.05. Are there reasons why the number should be exactly two or is it simply a convention that has the advantage of simplifying the math?

This question is quite old but I actually have an answer that doesn't appear here, and one that gives a compelling reason why (under some reasonable assumptions) squared error is correct, while any other power is incorrect.

Say we have some data $D = \langle(\mathbf{x}_1,y_1),(\mathbf{x}_2,y_2),...,(\mathbf{x}_n,y_n)\rangle$ and want to find the linear (or whatever) function $f$ that best predicts the data, in the sense that the probability density $p_f(D)$ for observing this data should be maximal with regard to $f$ (this is called the maximum likelihood estimation). If we assume that the data are given by $f$ plus a normally distributed error term with standard deviation $\sigma$, then $$p_f(D) = \prod_{i=1}^{n} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(y_i - f(\mathbf{x}_i))^2}{2\sigma^2}}.$$ This is equivalent to $$\frac{1}{\sigma^n(2\pi)^{n/2}}e^{-\frac{1}{2\sigma^2}\sum_{i=1}^{n} (y_i - f(\mathbf{x}_i))^2}.$$ So maximizing $p_f(D)$ is accomplished by minimizing $\sum_{i=1}^{n} (y_i - f(\mathbf{x}_i))^2$, that is, the sum of the squared error terms.

• That seems circular, why should you assume a normally distributed error term?
– Joe
Mar 13, 2018 at 6:18
• @Joe You shouldn't always, but if the only thing you know about the error term is that it has a mean of 0 and a finite expected absolute value, then that's the maximum-entropy assumption, so it can stand in for whatever unknown error function you actually have. If you have additional information about the error distribution then I suppose you could use it and find a more accurate maximum likelihood estimator.
– user91730
Apr 3, 2018 at 18:53
• " if the only thing you know about the error term is that it has a mean of 0 and a finite expected absolute value, then that's the maximum-entropy assumption" - every derivation of maximum entropy distributions I have seen derives the Laplace distribution as the maxent distribution for a (known) finite expected absolute value, whereas the Gaussian is the maxent for a (known) finite expected squared absolute value, see as one example stats.stackexchange.com/questions/82410/… do you have citations that disagree?
– Joe
Apr 4, 2018 at 19:29
• You know, I don't. I'll assume you're right. (Though I can't figure out how to edit my comment for some reason)
– user91730
Apr 6, 2018 at 13:02

There's no reason you couldn't try to minimize norms other than x^2, there have been entire books written on quantile regression, for instance, which is more or less minimizing |x| if you're working with the median. It's just generally harder to do and, depending on the error model, may not give good estimators (depending on whether that means low-variance or unbiased or low MSE estimators in the context).

As for why we prefer integer moments over real-number-valued moments, the main reason is likely that while integer powers of real numbers always result in real numbers, non-integer powers of negative real numbers create complex numbers, thus requiring the use of an absolute value. In other words, while the 3rd moment of a real-valued random variable is real, the 3.2nd moment is not necessarily real, and so causes interpretation problems.

Other than that...

1. Analytical expressions for the integer moments of random variables are typically much easier to find than real-valued moments, be it by generating functions or some other method. Methods to minimize them are thus easier to write.
2. The use of integer moments leads to expressions that are more tractable than real-valued moments.
3. I can't think of a compelling reason that (for instance) the 1.95th moment of the absolute value of X would provide better fitting properties than (for instance) the 2nd moment of X, although that could be interesting to investigate
4. Specific to the L2 norm (or squared error), it can be written via dot products, which can lead to vast improvements in speed of computation. It's also the only Lp space that's a Hilbert space, which is a nice feature to have.

In ordinary least squares, the solution to (A'A)^(-1) x = A'b minimizes squared error loss, and is the maximum likelihood solution.

So, largely because the math was easy in this historic case.

But generally people minimize many different loss functions, such as exponential, logistic, cauchy, laplace, huber, etc. These more exotic loss functions generally require a lot of computational resources, and don't have closed form solutions (in general), so they're only starting to become more popular now.

• +1 for introducing the idea of loss. (But aren't "exponential", etc., distributions, not loss functions?) Historically linear loss was the first approach formally developed, in 1750, and there was a straightforward geometric solution available for it. I believe Laplace established the relationship between this and the double-exponential distribution in an 1809 publication (for which the MLE will minimize absolute error, not squared error). Thus squared loss is not uniquely distinguished by the criteria of having an MLE and being mathematically easy.
– whuber
Nov 27, 2010 at 21:58
• They're both distributions and loss functions in different contexts.
– Joe
Dec 28, 2010 at 21:36
• I pressed enter too quickly on the previous reply - exponential loss is widely associated with boosting (see Friedman Hastie and Tibshirani's Statistical View of Boosting), where it's a loss rather than a distribution, logistic regression coresponds to log loss, laplace is a distribution but corresponds to absolute value loss - so for the most part I was being extremely sloppy, thanks for pointing it out. But while L1 loss has a geometric solution, it isn't analytically closed form, so I would hardly call its solution easy.
– Joe
Dec 28, 2010 at 21:49

We try to minimize the variance that is left within descriptors. Why variance? Read this question; this also comes together with the (mostly silent) assumption that errors are normally distributed.

Extension:
My understanding is that because we are trying to minimise errors, we need to find a way of not getting ourselves in a situation where the sum of the negative difference in errors is equal to the sum of the positive difference in errors but we haven't found a good fit. We do this by squaring the sum of the difference in errors which means the negative and positive difference in errors both become positive ($-1\times-1 = 1$). If we raised $x$ to the power of anything other than a positive integer we wouldn't address this problem because the errors would not have the same sign, or if we raised to the power of something that isn't an integer we'd enter the realms of complex numbers.