# Bias towards natural numbers in the case of least squares

Why do we seek to minimize x^2 instead of minimizing |x|^1.95 or |x|^2.05. Are there reasons why the number should be exactly two or is it simply a convention that has the advantage of simplifying the math?

-

This question is quite old but I actually have an answer that doesn't appear here, and one that gives a compelling reason why (under some reasonable assumptions) squared error is correct, while any other power is incorrect.

Say we have some data $D = \langle(\mathbf{x}_1,y_1),(\mathbf{x}_2,y_2),...,(\mathbf{x}_n,y_n)\rangle$ and want to find the linear (or whatever) function $f$ that best predicts the data, in the sense that the probability density $p_f(D)$ for observing this data should be maximal with regard to $f$ (this is called the maximum likelihood estimation). If we assume that the data are given by $f$ plus a normally distributed error term with standard deviation $\sigma$, then $$p_f(D) = \prod_{i=1}^{n} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(y_i - f(\mathbf{x}_i))^2}{2\sigma^2}}.$$ This is equivalent to $$\frac{1}{\sigma^n(2\pi)^{n/2}}e^{-\frac{1}{2\sigma^2}\sum_{i=1}^{n} (y_i - f(\mathbf{x}_i))^2}.$$ So maximizing $p_f(D)$ is accomplished by minimizing $\sum_{i=1}^{n} (y_i - f(\mathbf{x}_i))^2$, that is, the sum of the squared error terms.

-
A nice answer! I suggest you mention the term "maximum likelihood estimation" to the answer. – Silverfish Oct 10 '15 at 13:22
Not a bad idea, I think I'll do that. – Ian Oct 10 '15 at 13:31

There's no reason you couldn't try to minimize norms other than x^2, there have been entire books written on quantile regression, for instance, which is more or less minimizing |x| if you're working with the median. It's just generally harder to do and, depending on the error model, may not give good estimators (depending on whether that means low-variance or unbiased or low MSE estimators in the context).

As for why we prefer integer moments over real-number-valued moments, the main reason is likely that while integer powers of real numbers always result in real numbers, non-integer powers of negative real numbers create complex numbers, thus requiring the use of an absolute value. In other words, while the 3rd moment of a real-valued random variable is real, the 3.2nd moment is not necessarily real, and so causes interpretation problems.

Other than that...

1. Analytical expressions for the integer moments of random variables are typically much easier to find than real-valued moments, be it by generating functions or some other method. Methods to minimize them are thus easier to write.
2. The use of integer moments leads to expressions that are more tractable than real-valued moments.
3. I can't think of a compelling reason that (for instance) the 1.95th moment of the absolute value of X would provide better fitting properties than (for instance) the 2nd moment of X, although that could be interesting to investigate
4. Specific to the L2 norm (or squared error), it can be written via dot products, which can lead to vast improvements in speed of computation. It's also the only Lp space that's a Hilbert space, which is a nice feature to have.
-

We try to minimize the variance that is left within descriptors. Why variance? Read this question; this also comes together with the (mostly silent) assumption that errors are normally distributed.

Extension:

1. For variances, we have this nice "law" that the sum of variances is equal to the variance of sum, for uncorrelated samples. If we assume that the error is not correlated with the case, minimizing residual of squares will work straightforward to maximizing explained variance, what is maybe a not-so-good but still popular quality measure.

2. If we assume normality of an error, least squares error estimator is a maximal likelihood one.

-
The answer in that other thread don't really explain why 2 is a better value than other values that are very near to 2 but are no natural numbers. – Christian Jul 21 '10 at 0:04
I think it does; still I'll try to extend the answer. – mbq Jul 21 '10 at 0:21
So, if the errors are not normally distributed, but for instance according to another Lévy-stable distribution, it might pay off to use an exponent different from 2? – Raskolnikov Nov 28 '10 at 18:28
Remember, the normal distribution is the most "cautious" one for known variance (because is has maximum entropy among all densities with fixed variance). It leaves the most to be said by the data. Or put another way, for "large" data sets with the same variance, "you" have to "try" incredibly hard to get a distribution which is different from a normal. – probabilityislogic Feb 10 '11 at 12:16

In ordinary least squares, the solution to (A'A)^(-1) x = A'b minimizes squared error loss, and is the maximum likelihood solution.

So, largely because the math was easy in this historic case.

But generally people minimize many different loss functions, such as exponential, logistic, cauchy, laplace, huber, etc. These more exotic loss functions generally require a lot of computational resources, and don't have closed form solutions (in general), so they're only starting to become more popular now.

-
+1 for introducing the idea of loss. (But aren't "exponential", etc., distributions, not loss functions?) Historically linear loss was the first approach formally developed, in 1750, and there was a straightforward geometric solution available for it. I believe Laplace established the relationship between this and the double-exponential distribution in an 1809 publication (for which the MLE will minimize absolute error, not squared error). Thus squared loss is not uniquely distinguished by the criteria of having an MLE and being mathematically easy. – whuber Nov 27 '10 at 21:58
They're both distributions and loss functions in different contexts. – Joe Dec 28 '10 at 21:36
I pressed enter too quickly on the previous reply - exponential loss is widely associated with boosting (see Friedman Hastie and Tibshirani's Statistical View of Boosting), where it's a loss rather than a distribution, logistic regression coresponds to log loss, laplace is a distribution but corresponds to absolute value loss - so for the most part I was being extremely sloppy, thanks for pointing it out. But while L1 loss has a geometric solution, it isn't analytically closed form, so I would hardly call its solution easy. – Joe Dec 28 '10 at 21:49

My understanding is that because we are trying to minimise errors, we need to find a way of not getting ourselves in a situation where the sum of the negative difference in errors is equal to the sum of the positive difference in errors but we haven't found a good fit. We do this by squaring the sum of the difference in errors which means the negative and positive difference in errors both become positive ($-1\times-1 = 1$). If we raised $x$ to the power of anything other than a positive integer we wouldn't address this problem because the errors would not have the same sign, or if we raised to the power of something that isn't an integer we'd enter the realms of complex numbers.

-