Bias towards natural numbers in the case of least squares Why do we seek to minimize x^2 instead of minimizing |x|^1.95 or |x|^2.05.
Are there reasons why the number should be exactly two or is it simply a convention that has the advantage of simplifying the math?
 A: In ordinary least squares, the solution to (A'A)^(-1) x = A'b minimizes squared error loss, and is the maximum likelihood solution.
So, largely because the math was easy in this historic case.
But generally people minimize many different loss functions, such as exponential, logistic, cauchy, laplace, huber, etc.  These more exotic loss functions generally require a lot of computational resources, and don't have closed form solutions (in general), so they're only starting to become more popular now.
A: We try to minimize the variance that is left within descriptors. Why variance? Read this question; this also comes together with the (mostly silent) assumption that errors are normally distributed.
Extension:
Two additional arguments:  


*

*For variances, we have this nice "law" that the sum of variances is equal to the variance of sum, for uncorrelated samples. If we assume that the error is not correlated with the case, minimizing residual of squares will work straightforward to maximizing explained variance, what is maybe a not-so-good but still popular quality measure.  

*If we assume normality of an error, least squares error estimator is a maximal likelihood one.
A: This question is quite old but I actually have an answer that doesn't appear here, and one that gives a compelling reason why (under some reasonable assumptions) squared error is correct, while any other power is incorrect.
Say we have some data $D = \langle(\mathbf{x}_1,y_1),(\mathbf{x}_2,y_2),...,(\mathbf{x}_n,y_n)\rangle$ and want to find the linear (or whatever) function $f$ that best predicts the data, in the sense that the probability density $p_f(D)$ for observing this data should be maximal with regard to $f$ (this is called the maximum likelihood estimation). If we assume that the data are given by $f$ plus a normally distributed error term with standard deviation $\sigma$, then
$$p_f(D) = \prod_{i=1}^{n} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(y_i - f(\mathbf{x}_i))^2}{2\sigma^2}}.$$
This is equivalent to
$$\frac{1}{\sigma^n(2\pi)^{n/2}}e^{-\frac{1}{2\sigma^2}\sum_{i=1}^{n} (y_i - f(\mathbf{x}_i))^2}.$$
So maximizing $p_f(D)$ is accomplished by minimizing $\sum_{i=1}^{n} (y_i - f(\mathbf{x}_i))^2$, that is, the sum of the squared error terms.
A: There's no reason you couldn't try to minimize norms other than x^2, there have been entire books written on quantile regression, for instance, which is more or less minimizing |x| if you're working with the median.  It's just generally harder to do and, depending on the error model, may not give good estimators (depending on whether that means low-variance or unbiased or low MSE estimators in the context).  
As for why we prefer integer moments over real-number-valued moments, the main reason is likely that while integer powers of real numbers always result in real numbers, non-integer powers of negative real numbers create complex numbers, thus requiring the use of an absolute value.  In other words, while the 3rd moment of a real-valued random variable is real, the 3.2nd moment is not necessarily real, and so causes interpretation problems.
Other than that...


*

*Analytical expressions for the integer moments of random variables are typically much easier to find than real-valued moments, be it by generating functions or some other method.  Methods to minimize them are thus easier to write.

*The use of integer moments leads to expressions that are more tractable than real-valued moments.

*I can't think of a compelling reason that (for instance) the 1.95th moment of the absolute value of X would provide better fitting properties than (for instance) the 2nd moment of X, although that could be interesting to investigate

*Specific to the L2 norm (or squared error), it can be written via dot products, which can lead to vast improvements in speed of computation.  It's also the only Lp space that's a Hilbert space, which is a nice feature to have.

A: My understanding is that because we are trying to minimise errors, we need to find a way of not getting ourselves in a situation where the sum of the negative difference in errors is equal to the sum of the positive difference in errors but we haven't found a good fit. We do this by squaring the sum of the difference in errors which means the negative and positive difference in errors both become positive ($-1\times-1 = 1$). If we raised $x$ to the power of anything other than a positive integer we wouldn't address this problem because the errors would not have the same sign, or if we raised to the power of something that isn't an integer we'd enter the realms of complex numbers. 
