# Why square the difference instead of taking the absolute value in standard deviation?

In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute value of the difference instead and get the expected value (mean) of those, and wouldn't that also show the variation of the data? The number is going to be different from square method (the absolute-value method will be smaller), but it should still show the spread of data. Anybody know why we take this square approach as a standard?

The definition of standard deviation:

$\sigma = \sqrt{E\left[\left(X - \mu\right)^2\right]}.$

Can't we just take the absolute value instead and still be a good measurement?

$\sigma = E\left[|X - \mu|\right]$

-
In a way, the measurement you proposed is widely used in case of error (model quality) analysis -- then it is called MAE, "mean absolute error". –  mbq Jul 19 '10 at 21:30
In accepting an answer it seems important to me that we pay attention to whether the answer is circular. The normal distribution is based on these measurements of variance from squared error terms, but that isn't in and of itself a justification for using (X-M)^2 over |X-M|. –  rpierce Jul 20 '10 at 7:59
Do you think the term standard means this is THE standard today ? Isn't it like asking why principal component are "principal" and not secondary ? –  robin girard Jul 23 '10 at 21:44
My understanding of this question is that it could be shorter just be something like: what is the difference between the MAE and the RMSE ? otherwise it is difficult to deal with. –  robin girard Jul 24 '10 at 6:08
Related question: stats.stackexchange.com/q/354/919 ("Bias towards natural numbers in the case of least squares.") –  whuber Nov 27 '10 at 21:53
show 5 more comments

If the goal of the standard deviation is to summarise the spread of a symmetrical data set (i.e. in general how far each datum is from the mean), then we need a good method of defining how to measure that spread.

The benefits of squaring include:

• Squaring always gives a positive value, so the sum will not be zero.
• Squaring emphasizes larger differences - a feature that turns out to be both good and bad (think of the effect outliers have).

Squaring however does have a problem as a measure of spread and that is that the units are all squared, where as we'd might prefer the spread to be in the same units as the original data (think of squared pounds or squared dollars or squared apples). Hence the square root allows us to return to the original units.

I suppose you could say that absolute difference assigns equal weight to the spread of data where as squaring emphasises the extremes. Technically though, as others have pointed out, squaring makes the algebra much easier to work with and offers properties that the absolute method does not (for example, the variance is equal to the expected value of the square of the distribution minus the square of the mean of the distribution)

It's important to note however that there's no reason you couldn't take the absolute difference if that is your preference on how you wish to view 'spread' (sort of how some people see 5% as some magical thresh hold for p-values, when in fact it's situation dependent). Indeed, there are in fact several competing methods for measuring spread.

My view is to use the squared values because I like to think of how it relates to the Pythagorean Theorem of Statistics: c = sqrt(a^2 + b^2) ...this also helps me remember that when working with independent random variables, variances add, standard deviations don't. But that's just my personal subjective preference.

An much more indepth analysis can be read here.

-
"Squaring always gives a positive value, so the sum will not be zero." and so does absolute values. –  robin girard Jul 22 '10 at 9:54
@robin girard: That is correct, hence why I preceded that point with "The benefits of squaring include". I wasn't implying that anything about absolute values in that statement. I take your point though, I'll consider removing/rephrasing it if others feel it is unclear. –  Tony Breyal Jul 22 '10 at 13:19
Much of the field of robust statistics is an attempt to deal with the excessive sensitivity to outliers that that is a consequence of choosing the variance as a measure of data spread (technically scale or dispersion). en.wikipedia.org/wiki/Robust_statistics –  Thylacoleo Aug 13 '10 at 5:15
Thank you for the link to that analysis –  Jack Aidley Jan 23 at 14:03

The squared difference has nicer mathematical properties; it's continuously differentiable (nice when you want to minimize it), it's a sufficient statistic for the Gaussian distribution, and it's (a version of) the L2 norm which comes in handy for proving convergence and so on.

The mean absolute deviation (the absolute value notation you suggest) is also used as a measure of dispersion, but it's not as "well-behaved" as the squared error.

-
said "it's continuously differentiable (nice when you want to minimize it)" do you mean that the absolute value is difficult to optimize ? –  robin girard Jul 23 '10 at 21:40
@robin: while the absolute value function is continuous everywhere, its first derivative is not (at x=0). This makes analytical optimization more difficult. –  Vince Jul 23 '10 at 23:59
Yeah, finding quantiles in general (which includes optimizing absolute values) tends to churn up linear programming type problems, which -- while they're certainly tractable numerically -- can get fiddly. They typically don't have an analytical closed-form solution, and are a bit slower and a bit more difficult to implement than least-square-type solutions. –  Rich Jul 24 '10 at 2:55
I do not agree with this. First, theoretically, the problem may be of different nature (because of the discontinuity) but not necessarily harder (for example the median is easely shown to be arginf_m E[|Y-m|]). Second, practically, using a L1 norm (absolute value) rather than a L2 norm makes it piecewise linear and hence at least not more difficult. Quantile regression and its multiple variante is an example of that. –  robin girard Jul 24 '10 at 6:01
Yes, but finding the actual number you want, rather than just a descriptor of it, is easier under squared error loss. Consider the 1 dimension case; you can express the minimizer of the squared error by the mean: O(n) operations and closed form. You can express the value of the absolute error minimizer by the median, but there's not a closed-form solution that tells you what the median value is; it requires a sort to find, which is something like O(n log n). Least squares solutions tend to be a simple plug-and-chug type operation, absolute value solutions usually require more work to find. –  Rich Jul 24 '10 at 9:10

One way you can think of this is that standard deviation is similar to a "distance from the mean".

Compare this to distances in euclidean space - this gives you the true distance, where what you suggested (which, btw, is the absolute deviation) is more like a manhattan distance calculation.

-
Nice analogy of euclidean space! –  c4il Jul 19 '10 at 21:38
Yeah. Great analogy. –  Daniel Rodriguez Oct 31 '11 at 4:10
Except that in one dimension the $l_1$ and $l_2$ norm are the same thing, aren't they? –  naught101 Mar 29 '12 at 5:20
@naught101: It's not one dimension, but rather $n$ dimensions where $n$ is the number of samples. The standard deviation and the absolute deviation are (scaled) $l_2$ and $l_1$ distances respectively, between the two points $(x_1, x_2, \dots, x_n)$ and $(\mu, \mu, \dots, \mu)$ where $\mu$ is the mean. –  ShreevatsaR Nov 16 '12 at 7:21

Squaring the difference from the mean has a couple of reasons.

• Variance is defined as the 2nd moment of the deviation (the R.V here is (x-$\mu$) ) and thus the square as moments are simply the expectations of higher powers of the random variable.

• Having a square as opposed to the absolute value function gives a nice continuous and differentiable function (absolute value is not differentiable at 0) - which makes it the natural choice, especially in the context of estimation and regression analysis.

• The squared formulation also naturally falls out of parameters of the Normal Distribution.

-

The reason that we calculate standard deviation instead of absolute error is that we are assuming error to be normally distributed. It's a part of the model.

Suppose you were measuring very small lengths with a ruler, then standard deviation is a bad metric for error because you know you will never accidentally measure a negative length. A better metric would be one to help fit a Gamma distribution to your measurements:

$E(\log(x)) - \log(E(x))$

Like st. dev., this is also non-negative and differentiable, but it is a better error statistic for this problem.

-
I like your answer. The sd is not always the best statistic. –  RockScience Nov 25 '10 at 3:03

The answer that best satisfied me is that it falls out naturally from the generalization of a sample to n-dimensional euclidean space. It's certainly debatable whether that's something that should be done, but in any case:

Assume your $n$ measurements $X_i$ are each an axis in $\mathbb R^n$. Then your data $x_i$ define a point $\bf x$ in that space. Now you might notice that the data are all very similar to each other, so you can represent them with a single location parameter $\mu$ that is constrained to lie on the line defined by $X_i=\mu$. Projecting your datapoint onto this line gets you $\hat\mu=\bar x$, and the distance from the projected point $\hat\mu\bf 1$ to the actual datapoint is $\sqrt{\frac{n-1} n}\hat\sigma=\|\bf x-\hat\mu\bf 1\|$.

This approach also gets you a geometric interpretation for correlation, $\hat\rho=\cos \angle(\vec{\bf\tilde x},\vec{\bf\tilde y})$.

-
This is correct and appealing. However, in the end it appears only to rephrase the question without actually answering it: namely, why should we use the Euclidean (L2) distance? –  whuber Nov 24 '10 at 21:07
That is indeed an excellent question, left unanswered. I used to feel strongly that the use of L2 is unfounded. After having studied a little statistics, I saw the analytic niceties, and since then have revised my viewpoint into "if it really matters, you're probably in deep water already, and if not, easy is nice". I don't know measure theory yet, and worry that analysis rules there too - but I've noticed some new interest in combinatorics, so perhaps new niceties have been/will be found. –  sesqu Nov 24 '10 at 21:39
@sesqu Standard deviations did not become commonplace until Gauss in 1809 derived his eponymous deviation using squared error, rather than absolute error, as a starting point. However, what pushed them over the top (I believe) was Galton's regression theory (at which you hint) and the ability of ANOVA to decompose sums of squares--which amounts to a restatement of the Pythagorean Theorem, a relationship enjoyed only by the L2 norm. Thus the SD became a natural omnibus measure of spread advocated in Fisher's 1925 "Statistical Methods for Research Workers" and here we are, 85 years later. –  whuber Nov 24 '10 at 21:56
(+1) Continuing in @whuber's vein, I would bet that had Student published a paper in 1908 entitled, "Probable Error of the Mean - Hey, Guys, Check Out That MAE in the Denominator!" then statistics would have an entirely different face by now. Of course, he didn't publish a paper like that, and of course he couldn't have, because the MAE doesn't boast all the nice properties that S^2 has. One of them (related to Student) is its independence of the mean (in the normal case), which of course is a restatement of orthogonality, which gets us right back to L2 and the inner product. –  G. Jay Kerns Nov 25 '10 at 3:38

Yet another reason (in addition to the excellent ones above) comes from Fisher himself, who showed that the standard deviation is more "efficient" than the absolute deviation. Here, efficient has to do with how much a statistic will fluctuate in value on different samplings from a population. If your population is normally distributed, the standard deviation of various samples from that population will, on average, tend to give you values that are pretty similar to each other, whereas the absolute deviation will give you numbers that spread out a bit more. Now, obviously this is in ideal circumstances, but this reason convinced a lot of people (along with the math being cleaner), so most people worked with standard deviations.

-
Your argument depends on the data being normally distributed. If we assume the population to have a "double exponential" distribution, then the absolute deviation is more efficient (in fact it is a sufficient statistic for the scale) –  probabilityislogic Jul 16 '11 at 5:08
Yes, as I stated, "if your population is normally distributed." –  Eric Suh Sep 8 '11 at 19:49

Just so people know, there is a Math Overflow question on the same topic.

Why-is-it-so-cool-to-square-numbers-in-terms-of-finding-the-standard-deviation

The take away message is that using the square root of the variance leads to easier maths. A similar response is given by Rich and Reed above.

-

There are many reasons; probably the main is that it works well as parameter of normal distribution.

-
I agree. Standard deviation is the right way to measure dispersion if you assume normal distribution. And a lot of distributions and real data are an approximately normal. –  Łukasz Lew Jul 20 '10 at 14:40
I don't think you should say "natural parameter": the natural parameters of the normal distribution are mean and mean times precision. (en.wikipedia.org/wiki/Natural_parameter) –  Neil G Mar 12 '12 at 7:40
@NeilG Good point; I was thinking about "casual" meaning here. I'll think about some better word. –  mbq Mar 12 '12 at 10:41

I think the contrast between using absolute deviations and squared deviations becomes clearer once you move beyond a single variable and think about linear regression. There's a nice discussion at http://en.wikipedia.org/wiki/Least_absolute_deviations, particularly the section "Contrasting Least Squares with Least Absolute Deviations" , which links to some student exercises with a neat set of applets at http://www.math.wpi.edu/Course_Materials/SAS/lablets/7.3/73_choices.html .

To summarise, least absolute deviations is more robust to outliers than ordinary least squares, but it can be unstable (small change in even a single datum can give big change in fitted line) and doesn't always have a unique solution - there can be a whole range of fitted lines. Also least absolute deviations requires iterative methods, while ordinary least squares has a simple closed-form solution, though that's not such a big deal now as it was in the days of Gauss and Legendre, of course.

-

Naturally you can describe dispersion of a distribution in any way meaningful (absolute deviation, quantiles, etc.).

One nice fact is that the variance is the second central moment, and every distribution is uniquely described by its moments if they exist. Another nice fact is that the variance is much more tractable mathematically than any comparable metric. Another fact is that the variance is one of two parameters of the normal distribution for the usual parametrization, and the normal distribution only has 2 non-zero central moments which are those two very parameters. Even for non-normal distributions it can be helpful to think in a normal framework.

As I see it, the reason the standard deviation exists as such is that in applications the square-root of the variance regularly appears (such as to standardize a random varianble), which necessitated a name for it.

-

Estimating the standard deviation of a distribution requires to choose a distance.
Any of the following distance can be used:

$d_{n}((X)_{i=1..I},\mu)=(\sum \left | X-\mu \right |^n)^{1/n}$

We usually use the natural euclidean distance (n=2), which is the one everybody uses in daily life. The distance that you propose is the one with $n=1$.
Both are good candidates but they are different.

One could decide to use $n=3$ as well.

I am not sure that you will like my answer, my point contrary to others is not to demonstrate that $n=2$ is better. I think that if you want to estimate the standard deviation of a distribution, you can absolutely use a different distance.

-
Did you mean $n=1$ instead of the (undefined) $n=0$? –  whuber Jan 5 '11 at 3:25
Yes indeed, thx –  RockScience Jan 5 '11 at 3:40

It depends on what you are talking about when you say "spread of the data". To me this could mean two things:

1. The width of a sampling distribution
2. The accuracy of a given estimate

For point 1) there is no particular reason to use the standard deviation as a measure of spread, except for when you have a normal sampling distribution. The measure $E(|X-\mu|)$ is a more appropriate measure in the case of a Laplace Sampling distribution. My guess is that the standard deviation gets used here because of intuition carried over from point 2). Probably also due to the success of least squares modelling in general, for which the standard deviation is the appropriate measure. Probably also because calculating $E(X^2)$ is generally easier than calculating $E(|X|)$ for most distributions.

Now, for point 2) there is a very good reason for using the variance/standard deviation as the measure of spread, in one particular, but very common case. You can see it in the Laplace approximation to a posterior. With Data $D$ and prior information $I$, write the posterior for a parameter $\theta$ as:

$$p(\theta|DI)=\frac{\exp\left(h(\theta)\right)}{\int \exp\left(h(t)\right)dt}\;\;\;\;\;\;h(\theta)\equiv\log[p(\theta|I)p(D|\theta I)]$$

I have used $t$ as a dummy variable to indicate that the denominator does not depend on $\theta$. If the posterior has a single well rounded maximum (i.e. not too close to a "boundary"), we can taylor expand the log probability about its maximum $\theta_{max}$. If we take the first two terms of the taylor expansion we get (using prime for differentiation):

$$h(\theta)\approx h(\theta_{max})+(\theta_{max}-\theta)h'(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$

But we have here that because $\theta_{max}$ is a "well rounded" maximum, $h'(\theta_{max})=0$, so we have:

$$h(\theta)\approx h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$

If we plug in this approximation we get:

$$p(\theta|DI)\approx\frac{\exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$

$$=\frac{\exp\left(\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$

Which, but for notation is a normal distribution, with mean equal to $E(\theta|DI)\approx\theta_{max}$, and variance equal to

$$V(\theta|DI)\approx \left[-h''(\theta_{max})\right]^{-1}$$

($-h''(\theta_{max})$ is always positive because we have a well rounded maximum). So this means that in "regular problems" (which is most of them), the variance is the fundamental quantity which determines the accuracy of estimates for $\theta$. So for estimates based on a large amount of data, the standard deviation makes a lot of sense theoretically - it tells you basically everything you need to know. Essentially the same argument applies (with same conditions required) in multi-dimensional case with $h''(\theta)_{jk}=\frac{\partial h(\theta)}{\partial \theta_j \partial \theta_k}$ being a Hessian matrix. The diagonal entries are also essentially variances here too.

The frequentist using the method of maximum likelihood will come to essentially the same conclusion because the MLE tends to be a weighted combination of the data, and for large samples the Central Limit Theorem applies and you basically get the same result if we take $p(\theta|I)=1$ but with $\theta$ and $\theta_{max}$ interchanged: $$p(\theta_{max}|\theta)\approx N\left(\theta,\left[-h''(\theta_{max})\right]^{-1}\right)$$ (see if you can guess which paradigm I prefer :P ). So either way, in parameter estimation the standard deviation is an important theoretical measure of spread.

-

Because squares can allow use of many other mathematical operations or functions more easily than absolute values.

Example: squares can be integrated, differentiated, can be used in trigonometric, logarithmic and other functions, with ease.

-
I wonder if there is a self fulfilling profecy here. We get –  probabilityislogic Mar 13 '12 at 12:04

$\newcommand{\var}{\operatorname{var}}$ Variances are additive: for independent random variables $X_1,\ldots,X_n$, $$\var(X_1+\cdots+X_n)=\var(X_1)+\cdots+\var(X_n).$$

Notice what this makes possible: Say I toss a fair coin 900 times. What's the probability that the number of heads I get is between 440 and 455 inclusive? Just find the expected number of heads ($450$), and the variance of the number of heads ($225=15^2$), then find the probability with a normal (or Gaussian) distribution with expectation $450$ and standard deviation $15$ is between $439.5$ and $455.5$. Abraham de Moivre did this with coin tosses in the 18th century, thereby first showing that the bell-shaped curve is worth something.

-
Are mean absolute deviations not additive in the same way as variances? –  rpierce Feb 9 at 23:30
No, they're not. –  Michael Hardy Feb 10 at 18:14