18
$\begingroup$

As I understand UK Schools teach that the Standard Deviation is found using:

alt text

whereas US Schools teach:

alt text

(at a basic level anyway).

This has caused a number of my students problems in the past as they have searched on the Internet, but found the wrong explanation.

Why the difference?

With simple datasets say 10 values, what degree of error will there be if the wrong method is applied (eg in an exam)?

$\endgroup$
5
  • 4
    $\begingroup$ I am not sure if characterizing one or the other as the 'wrong' formula is the way to understand the issue. It is just that the second one is 'better' in in the sense that it is an unbiased estimator of the true standard deviation. So, if you care about unbiased estimates then the second one is 'better'/'correct'. $\endgroup$
    – user28
    Jul 19 '10 at 19:51
  • $\begingroup$ I was characterising the formula as "wrong" purely in the sense that in an exam if you use the formula which isn't proscribed by the syllabus you'll end up with the "wrong" answer. Plus if the values are not a sample of population per se then surely the first formula gives the more accurate value. $\endgroup$
    – Amos
    Jul 19 '10 at 20:05
  • 14
    $\begingroup$ Srikant, I don't think that the second one is an unbiased estimator. The square of it is an unbiased estimator of the true variance. However, Jensen's Inequality establishes that the expectation of a curvilinear function of a random variable is not the same as the function of the expectation of the random variable. Hence the second formula can't be an unbiased estimator of the true standard deviation. $\endgroup$ Jul 20 '10 at 5:28
  • $\begingroup$ For cross-reference: it was also asked @ m.SE ... $\endgroup$ Dec 22 '10 at 7:00
  • 4
    $\begingroup$ Any US school using the very popular elementary text by Freedman, Pisani, & Purves is using the first formula ($s_n$), so it seems incorrect to characterize this as a US vs. UK difference. $\endgroup$
    – whuber
    Dec 26 '12 at 18:46
20
$\begingroup$

The first formula is the population standard deviation and the second formula is the the sample standard deviation. The second formula is also related to the unbiased estimator of the variance - see wikipedia for further details.

I suppose (here) in the UK they don't make the distinction between sample and population at high school. They certainly don't touch concepts such as biased estimators.

$\endgroup$
3
  • 4
    $\begingroup$ Colin, an unbiased estimator of the standard deviation does not have a closed form representation in the general case. What does exist is the unbiased estimator of the <i>variance</i> (s<sup>2</sup> in this case). Noteworthy that both are consistent estimators of the population variance - and so by the continuous mapping theorem, are the two estimators of the standard deviations. A related point is that s<sub>n</sub><sup>2</sup> has a lower MSE than s<sup>2</sup>. The additional advantage from imposing unbiasedness is arguable. $\endgroup$
    – Relevance
    Jul 19 '10 at 20:30
  • $\begingroup$ @Tirthankar - very sloppy of me. I've altered the answer slightly. Thanks. $\endgroup$ Jul 19 '10 at 20:45
  • 2
    $\begingroup$ As far as I remember, I was taught the 'sample' calculation in GCSE maths and science (age 14-16) and the distinction between populations and samples and their associated variance measures was covered (though not in depth) at A-level (age 16-18). So I'm not sure this is a simple UK/US difference. $\endgroup$ Oct 19 '10 at 15:55
11
$\begingroup$

Because nobody has yet answered the final question--namely, to quantify the differences between the two formulas--let's take care of that.

For many reasons, it is appropriate to compare standard deviations in terms of their ratios rather than their differences. The ratio is

$$s_n / s = \sqrt{\frac{N-1}{N}} = \sqrt{1 - \frac{1}{N}} \approx 1 - \frac{1}{2N}.$$

The approximation can be viewed as truncating the (alternating) Taylor series for the square root, indicating the error cannot exceed $|\binom{1/2}{2}N^{-2}|$ = $1 / (8 N^2)$. This establishes that the approximation is more than good enough (for our purposes) once $N$ is $2$ or larger.

It is immediate that the two SD estimates are within (about) 10% of each other once $N$ exceeds $5$, within 5% once $N$ exceeds $10$, and so on. Clearly, for many purposes these discrepancies are so small that it does not matter which formula is used, especially when the SD is intended for describing the spread of data or for making semi-quantitative assessments or predictions (such as in employing the 68-95-99.7 rule of thumb). The discrepancies are even less important when comparing SDs, such as when comparing the spreads of two datasets. (When the datasets are equinumerous, the discrepancies effectively vanish altogether and both formulas lead to identical conclusions.) Arguably, these are the forms of reasoning we are trying to teach beginning students, so if the students are becoming concerned about which formula to use, that could be taken as a sign that the text or the class is failing to emphasize what is really important.

We might want to pay some attention to the case of very small $N$. Here, people may be using $t$ tests instead of $z$ tests, for instance. In that case, it is essential to employ whichever formula for the standard deviation is used by one's table or software. (This is not a matter of one formula being wrong or right; it's just a consistency requirement.) Most tables use $s$, not $s_n$: this is the one place in the elementary syllabus where the text and teacher need to be clear about which formula to use.

$\endgroup$
6
$\begingroup$

This is Bessel's Correction. The US version is showing the formula for the sample standard deviation, where the UK version above is the standard deviation of the sample.

$\endgroup$
5
$\begingroup$

I am not sure this is purely a US vs. British issue. The rest of this page is excerpted from a faq I wrote.(http://www.graphpad.com/faq/viewfaq.cfm?faq=1383).

How to compute the SD with n-1 in the denominator

  1. Compute the square of the difference between each value and the sample mean.

  2. Add those values up.

  3. Divide the sum by n-1. The result is called the variance.

  4. Take the square root to obtain the Standard Deviation.

Why n-1?

Why divide by n-1 rather than n when computing a standard deviation? In step 1, you compute the difference between each value and the mean of those values. You don't know the true mean of the population; all you know is the mean of your sample. Except for the rare cases where the sample mean happens to equal the population mean, the data will be closer to the sample mean than it will be to the true population mean. So the value you compute in step 2 will probably be a bit smaller (and can't be larger) than what it would be if you used the true population mean in step 1. To make up for this, divide by n-1 rather than n.v This is called Bessel's correction.

But why n-1? If you knew the sample mean, and all but one of the values, you could calculate what that last value must be. Statisticians say there are n-1 degrees of freedom.

When should the SD be computed with a denominator of n instead of n-1?

Statistics books often show two equations to compute the SD, one using n, and the other using n-1, in the denominator. Some calculators have two buttons.

The n-1 equation is used in the common situation where you are analyzing a sample of data and wish to make more general conclusions. The SD computed this way (with n-1 in the denominator) is your best guess for the value of the SD in the overall population.

If you simply want to quantify the variation in a particular set of data, and don't plan to extrapolate to make wider conclusions, then you can compute the SD using n in the denominator. The resulting SD is the SD of those particular values. It makes no sense to compute the SD this way if you want to estimate the SD of the population from which those points were drawn. It only makes sense to use n in the denominator when there is no sampling from a population, there is no desire to make general conclusions.

The goal of science is almost always to generalize, so the equation with n in the denominator should not be used. The only example I can think of where it might make sense is in quantifying the variation among exam scores. But much better would be to show a scatterplot of every score, or a frequency distribution histogram.

$\endgroup$
3
  • 1
    $\begingroup$ I wasn't suggesting it was, I was just curious as to why such a difference might have arisen, what sort of level of error following the wrong advice might give and whether there was a decent explanation of the difference I could give to my students. $\endgroup$
    – Amos
    Jul 19 '10 at 20:03
  • $\begingroup$ @harvey - the link is dead $\endgroup$
    – baxx
    Jan 11 '20 at 15:30
  • 1
    $\begingroup$ @baxx.. Thanks for pointing this out. Fixed. $\endgroup$ Jan 13 '20 at 0:55
3
$\begingroup$

Since N is the number of points in the data set, one could argue that by calculating the mean one has reduced the degree of freedom in the data set by one (since one introduced a dependency into the data set), so one should use N-1 when estimating the standard deviation from a data set for which one had to estimate the mean before.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.