Today I taught an introductory class of statistics and a student came up to me with a question, which I rephrase here as: "Why is the standard deviation defined as sqrt of variance and not as the sqrt of sum of squares over N?"

We define population variance: $\sigma^2=\frac{1}{N}\sum{(x_i-\mu)^2}$

And standard deviation: $\sigma=\sqrt{\sigma^2}=\frac{1}{\sqrt{N}}\sqrt{\sum{(x_i-\mu)^2}}$.

The interpretation we may give to $\sigma$ is that it gives the average deviation of units in the population from the population mean of $X$.

However, in the definition of the s.d. we divide the sqrt of the sum of squares through $\sqrt{N}$. The question the student raises is why we do not divide the sqrt of the sume of squares by $N$ instead. Thus we come to competing formula: $$\sigma_{new}=\frac{1}{N}\sqrt{\sum{(x_i-\mu)^2}}.$$ The student argued that this formula looks more like an "average" deviation from the mean than when dividing through $\sqrt{N}$ as in $\sigma$.

I thought this question is not stupid. I would like to give an answer to the student that goes further than saying that the s.d. is defined as sqrt of the variance which is the average squared deviaton. Put differently, why should the student use the correct formula and not follow her idea?

This question relates to an older thread and answers provided here. Answers there go in three directions:

  1. $\sigma$ is the root-mean-squared (RMS) deviation, not the "typical" deviation from the mean (i.e., $\sigma_{new}$). Thus, it is defined differently.
  2. It has nice mathematical properties.
  3. Furthermore, the sqrt would bring back "units" to their original scale. However, this would also be the case for $\sigma_{new}$, which divides by $N$ instead.

Both of points 1 and 2 are arguments in favour of the s.d. as RMS, but I do not see an argument against the use of $\sigma_{new}$. What would be the good arguments to convince introductory level students of the use of the average RMS distance $\sigma$ from the mean?

  • 2
    I think the very question "Why is the standard deviation defined as..." is hard to answer. Definitions are just arbitrary labeling conventions. They don't have to conform to why's. – ttnphns Sep 22 '14 at 18:18
  • "Why is the standard deviation defined as sqrt of variance and not as average of [the root of] sum of squares?" Might it be that what is inside brackets got somehow lost in the question? – ttnphns Sep 22 '14 at 18:21
  • 1
    But s.d. serves a series of purposes; there must be better motivation than that it is defined like that. That would be useful, especially in teaching undergraduates. I can imagine a motivation in the sense of Chebyshev's inequality (min. of proportion of cases in realm of +/- a constant factor of s.d.). – tomka Sep 22 '14 at 18:22
  • 2
    Can't answer because your Q is on hold, but try this: Imagine you observe values 1 and 3 in roughly equal proportions (toss a coin, $H=3$, $T=1$). A "typical distance" of observations from the mean should be something like 1. With your $\sqrt{SSE}/n$ formula, consider what happens to this measure of typical distance for $n$ very, very large. In each case $|x_i-\bar{x}|$ will be near 1, so their sum of squares will be near $n$. The numerator will be close to $\sqrt{n}$ so your formula would get smaller and smaller as $n$ increased, even though the typical distance from the mean wasn't changing. – Glen_b Sep 22 '14 at 23:05
  • 1
    @whuber I made another update and hope the point I make is clearer now. Note I am asking for teaching advice here besides asking a question on fundations of statistics. I am not suggesting an alternative formula, but gave an example from a classroom situation of a good question by a student to which I did not have an immediate answer. If you agree, I kindly request to release the question from hold now . – tomka Sep 23 '14 at 17:33
up vote 12 down vote accepted

There are at least three basic problems which can readily be explained to beginners:

  1. The "new" SD is not even defined for infinite populations. (One could declare it always to equal zero in such cases, but that would not make it any more useful.)

  2. The new SD does not behave the way an average should do under random sampling.

  3. Although the new SD can be used with all mathematical rigor to assess deviations from a mean (in samples and finite populations), its interpretation is unnecessarily complicated.

1. The applicability of the new SD is limited

Point (1) could be brought home, even to those not versed in integration, by pointing out that because the variance clearly is an arithmetic mean (of squared deviations), it has a useful extension to models of "infinite" populations for which the intuition of the existence of an arithmetic mean still holds. Therefore its square root--the usual SD--is perfectly well defined in such cases, too, and just as useful in its role as a (nonlinear reexpression of) a variance. However, the new SD divides that average by the arbitrarily large $\sqrt{N}$, rendering problematic its generalization beyond finite populations and finite samples: what should $1/\sqrt{N}$ be taken to equal in such cases?

2. The new SD is not an average

Any statistic worthy of the name "average" should have the property that it converges to the population value as the size of a random sample from the population increases. Any fixed multiple of the SD would have this property, because the multiplier would apply both to computing the sample SD and the population SD. (Although not directly contradicting the argument offered by Alecos Papadopoulos, this observation suggests that argument is only tangential to the real issues.) However, the "new" SD, being equal to $1/\sqrt{N}$ times the usual one, obviously converges to $0$ in all circumstances as the sample size $N$ grows large. Therefore, although for any fixed sample size $N$ the new SD (suitably interpreted) is a perfectly adequate measure of variation around the mean, it cannot justifiably be considered a universal measure applicable, with the same interpretation, for all sample sizes, nor can it correctly be called an "average" in any useful sense.

3. The new SD is complicated to interpret and use

Consider taking samples of (say) size $N=4$. The new SD in these cases is $1/\sqrt{N}=1/2$ times the usual SD. It therefore enjoys comparable interpretations, such as an analog of the 68-95-99 rule (about 68% of the data should lie within two new SDs of the mean, 95% of them within four new SDs of the mean, etc.; and versions of classical inequalities such as Chebychev's will hold (no more than $1/k^2$ of the data can lie more than $2k$ new SDs away from their mean); and the Central Limit Theorem can be analogously restated in terms of the new SD (one divides by $\sqrt{N}$ times the new SD in order to standardize the variable). Thus, in this specific and clearly constrained sense, there is nothing wrong with the student's proposal. The difficulty, though, is that these statements all contain--quite explicitly--factors of $\sqrt{N}=2$. Although there is no inherent mathematical problem with this, it certainly complicates the statements and interpretation of the most fundamental laws of statistics.


It is of note that Gauss and others originally parameterized the Gaussian distribution by $\sqrt{2}\sigma$, effectively using $\sqrt{2}$ times the SD to quantify the spread of a Normal random variable. This historical use demonstrates the propriety and effectiveness of using other fixed multiples of the SD in its stead.

  • Thank you - one question back (relating to your point 2): does $\frac{1}{\sqrt{N}}$ not converge to $0$ as $N$ grows large, whereas $\frac{1}{N}$ obviously does? – tomka Sep 23 '14 at 19:14
  • 2
    We're comparing the SD of the sample to $1/\sqrt{N}$ times the SD of the sample (the "new SD"). As $N$ grows large, the SD of the sample approaches a (usually) nonzero constant equal to the population SD. Therefore $1/\sqrt{N}$ times the sample SD converges to zero. – whuber Sep 23 '14 at 19:35
  • This is standard material--consult any rigorous textbook in mathematical statistics (which, to be fair, would not be accessible to most beginners). However, the results important for my answer follow from a weaker and intuitively obvious statement. Fix a number $A \gt 1$ and let $\sigma$ be the population SD. Consider the chance that the sample SD will lie between $\sigma/A$ and $A\sigma$. It suffices that this chance goes to zero as the sample size $N$ increases. This alone shows that $1/\sqrt{N}$ times the sample SD converges to $0$ almost surely, demonstrating point (2) in the answer. – whuber Sep 24 '14 at 15:58
  • +1, plus it is not scale-invariant etc, (a condition necessary for a moment of this form) – Nikos M. Sep 25 '14 at 23:58
  • @Nikos Thank you, but what is not scale invariant? Both $SD/\sqrt{N}$ and $SD$ change when the data are rescaled. – whuber Sep 26 '14 at 16:41

Assume that your sample contains only two realizations. I guess an intuitive measure of dispersion would be the average absolute deviation (AAD)

$$AAD = \frac 12 (|x_1-\bar x| + |x_2-\bar x|) = ...= \frac {|x_1-x_2|}{2}$$

So we would want other measures of dispersion at the same level of units of measurement to be "close" to the above.

The sample variance is defined as

$$\sigma^2=\frac{1}{2}[(x_1-\bar x)^2 + (x_2-\bar x)^2] = \frac 12 \left[\left(\frac {x_1-x_2}{2}\right)^2 + \left(\frac {x_2-x_1}{2}\right)^2\right]$$

$$=\frac 12 \left[\frac {(x_1-x_2)^2}{4} + \frac {(x_1-x_2)^2}{4}\right]=\frac 12 \frac {(x_1-x_2)^2}{2}$$

$$=\frac 12\cdot \frac {|x_1-x_2|^2}{2}$$

To return to the original units of measurement, if we did as the student wondered/suggested,we would obtain the measure, call it $q$

$$ q \equiv \frac 12\cdot \sqrt {\frac {|x_1-x_2|^2}{2}} = \frac 12 \frac {|x_1-x_2|}{\sqrt 2} = \frac 1{\sqrt 2} AAD < AAD$$

i.e. we would have "downplayed" the "intuitive" measure of dispersion, while if we have considered the standard deviation as defined,

$$SD \equiv \sqrt {\sigma^2} = \frac {|x_1-x_2|}{2} =AAD$$

Since we want to "stay as close as possible" to the intuitive measure, we should use $SD$.

ADDENDUM
Let's consider now a sample of size $n$ We have

$$n\cdot AAD = \sum_{i=1}^n |x_i-\bar x|$$

and

$$n \cdot \text{Var}(X) = \sum_{i=1}^n (x_i-\bar x)^2 = \sum_{i=1}^n |x_i-\bar x|^2$$

we can write the right-hand side of the variance expression as

$$\sum_{i=1}^n |x_i-\bar x|^2 = \left(\sum_{i=1}^n |x_i-\bar x|\right)^2 - \sum_{j\neq i} |x_i-\bar x||x_j-\bar x|$$

$$ = \left (n\cdot AAD\right)^2 - \sum_{j\neq i} |x_i-\bar x||x_j-\bar x|$$

Then the dispersion measure $q_n$ will be

$$q_n \equiv \frac 1n \left[n^2\cdot AAD^2 - \sum_{j\neq i} |x_i-\bar x||x_j-\bar x|\right]^{1/2}$$

$$= \left[AAD^2 - \frac 1{n^2} \sum_{j\neq i} |x_i-\bar x||x_j-\bar x|\right]^{1/2}$$

Now think informally: note that $\sum_{j\neq i} |x_i-\bar x||x_j-\bar x|$ contains $n^2-n$ terms, and so divided by $n^2$ will left us with "one term in the second power". But also "one term in the 2nd power" is what we have in $AAD^2$: this is a primitive way to "sense" why $q_n$ will tend to zero as $n$ grows large. On the other hand the Standard Deviation as defined would be

$$SD \equiv \frac 1{\sqrt n} \left[n^2\cdot AAD^2 - \sum_{j\neq i} |x_i-\bar x||x_j-\bar x|\right]^{1/2}$$

$$= \left[n\cdot AAD^2 - \frac 1{n} \sum_{j\neq i} |x_i-\bar x||x_j-\bar x|\right]^{1/2}$$

Continuing are informal thinking, the first term gives us $n$ "terms in the 2nd power", while the second term gives us $n-1$ "terms in the second power" . So we will be left eventually with one such term, as $n$ grows large, and then we will take its square root.
This does not mean that the Standard Deviation as defined will equal the Average Absolute Deviation in general (it doesn't), but it does show that it is suitably defined so as to be "on a par" with it for any $n$, as well as for the case when $n\rightarrow \infty$.

  • 1
    Although this answer is interesting, I believe there are more important, convincing, and rigorous explanations (of which I have offered only a few in my own answer: much more could be said, especially concerning the role of the SD in the Central Limit theorem and algebraic rules for computing SDs of sums of independent random variables). – whuber Sep 23 '14 at 18:13
  • 2
    @whuber Certainly. I just opted for a "the bell has rung" approach to destroy the student's intermission! – Alecos Papadopoulos Sep 23 '14 at 19:33

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.