Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

The formula for computing variance has $(n-1)$ in the denominator:

$\sigma^2 = \frac{\sum_{i=1}^N (x_i - \mu)^2}{n-1}$

I've always wondered why. However, reading and watching a few good videos about "why" it is, it seems, $(n-1)$ is a good unbiased estimator of the population variance. Whereas $n$ underestimates and $(n-2)$ overestimates the population variance.

What I'm curious to know, is that in the era of no computers how exactly was this choice made? Is there an actual mathematical proof proving this or was this purely empirical and statisticians made A LOT of calculations by hand to come up with the "best explanation" at the time?

Just how did statisticians come up with this formula in the early 19th century with the aid of computers? Manual or there is more to it than meets the eye?

share|improve this question
6  
I assume you mean to say "without the aid of computers". The answer is - perhaps unsurprisingly - by the use of algebra. The derivation is quite straightforward and in many places it's common for statistics students to derive it as an exercise/learn it as undergrads. –  Glen_b May 26 at 2:26
    
I think this gives a pretty good explanation: en.wikipedia.org/wiki/Variance#Sample_variance –  ExpectoPatronum May 26 at 10:55
5  
add comment

3 Answers 3

up vote 18 down vote accepted

The correction is called Bessel's correction and it has a mathematical proof. Personally, I was taught it the easy way: using $n-1$ is how you correct the bias of $E[\frac{1}{n}\sum_1^n(x_i - \bar x)^2]$ (see here).

You can also explain the correction based on the concept of degrees of freedom, simulation isn't strictly needed.

share|improve this answer
5  
Proof alternate #3 has a beautiful intuitive explanation that even a lay person can understand. The basic idea is that the sample mean is not the same as the population mean. Your observations are naturally going to be closer to the sample mean than the population mean, and this ends up underestimating those $(x_i - \mu)^2$ terms with $(x_i - \bar{x})^2$ terms. This is probably obvious to most people but I never thought about the "intuition" as to why the biased sample variance is biased until now. I only learned the formal proofs. –  MHH May 26 at 4:52
    
There is also a geometrical approach why to correct with n-1 (explained very nicely in Saville and Wood: Statistical Methods: The Geometric Approach). Put it shortly: A sample of n can be regarded as an n-dimensional data space. The sample point vectors add to a observed vector which can be decomposed to a model vector with p-dimension corresponding to p parameter and an error vector with n-p dimension. The corresponding Pythagorean breakup of the error vector has n-p squares which average is a measure for the variation. –  giordano May 27 at 16:42
add comment

According to Weisstein's World of Mathematics, it was first proved by Gauss in 1823. The reference is volume 4 of Gauss' Werke, which can be read at https://archive.org/details/werkecarlf04gausrich. The relevant pages seem to be 47-49. It seems that Gauss investigated the question and came up with a proof. I don't read Latin, but there is a German summary in the text. Pages 103-104 explain what he did (Edit: I added a rough translation):

Allein da man nicht berechtigt ist, die sichersten Werthe fuer die wahren Werthe selbst zu halten, so ueberzeugt man sich leicht, dass man durch dieses Verfahren allemal den wahrscheinlichsten und mittleren Fehler zu klein finden muss, und daher die gegebenen Resultaten eine groessere Genauigkeit beilegt, als sie wirklich besitzen. [But since one is not entitled to treat the most probable values as though they were the actual values, one can easily convince oneself that one must always find that the most probable error and the average error are too small, and that therefore the given results possess a greater accuracy than they really have.]

from which it would seem that it was well-known that the sample variance is a biased estimate of the population variance. The article goes on to say that the difference between the two is usually ignored because it's not important if the sample size is big enough. Then it says:

Der Verfasser hat daher diesen Gegenstand eine besondere Untersuchung unterworfen, die zu einem sehr Merkwuerdigen hoechst einfachen Resultate gefuehrt hat. Man braucht nemlich den nach dem angezeigten fahlerhaften Verfahren gefundenen mittleren Fehler, um ihn in die richtigen zu verwandeln, nur mit

$$\sqrt{\frac{\pi-\rho}{\pi}}$$

zu multiplicieren, wo $\pi$ die Anzahl der beobachtungen (number of observations) und $\rho$ die Anzahl der unbekannten groessen (number of unknowns) bedeutet. [The author has therefore made a special study of this object which has led to a very strange and extremely simple result. Namely, one needs only to multiply the average error found by the above erroneous process by (the given expression) to change it into the right one, where $\pi$ is the number of observations and $\rho$ is the number of unknown quantities.]

So if this is indeed the first time that the correction was found, then it seems that it was found by a clever calculation by Gauss, but people were already aware that some correction was required, so perhaps someone else could have found it empirically before this. Or possibly previous authors didn't care to derive the precise answer because they were working with fairly large data sets anyway.

Summary: manual, but people already knew that $n$ in the denominator wasn't quite right.

share|improve this answer
    
If someone could provide a translation of the German, that would be nice. I for one don't read German. –  Faheem Mitha Jun 10 at 23:52
    
Yes, Google Translate doesn't work so well because of my spelling errors! I'll add in an attempt at translation; it will be a good way of practising my German. –  Flounderer Jun 10 at 23:55
    
Thanks, Flounderer, I appreciate it. –  Faheem Mitha Jun 10 at 23:58
add comment

I've been looking for a derivation on CV that I could link you to (there are a number of links to proofs off-site, including at least one in answers here), but I haven't found one here on CV in a couple of searches, so for the sake of completeness, I'll give a simple one. If I find one as simple as this on CV to link to I may delete this one.

This takes $E(X^2)=\text{Var}(X) + E(X)^2$ as assumed knowledge, and assumes that the first few basic variance properties are known.

\begin{eqnarray} E[\sum_{i=1}^{n} (x_i-\bar x)^2] &=& E[\sum_{i=1}^{n} x_i^2-2\bar x\sum_{i=1}^{n} x_i+n\bar{x}^2]\\ &=& E[\sum_{i=1}^{n} x_i^2-n\bar{x}^2] \\ &=& n E[x_i^2]- n E[\bar{x}^2]\\ &=& n (\mu^2 + \sigma^2) - n(\mu^2+\sigma^2/n)\\ &=& (n-1) \sigma^2 \end{eqnarray}

[Most proofs I have seen are simple enough that Gauss (however he did it) probably found it pretty easy to prove.]

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.