68
$\begingroup$

The formula for computing variance has $(n-1)$ in the denominator:

$s^2 = \frac{\sum_{i=1}^N (x_i - \bar{x})^2}{n-1}$

I've always wondered why. However, reading and watching a few good videos about "why" it is, it seems, $(n-1)$ is a good unbiased estimator of the population variance. Whereas $n$ underestimates and $(n-2)$ overestimates the population variance.

What I'm curious to know, is that in the era of no computers how exactly was this choice made? Is there an actual mathematical proof proving this or was this purely empirical and statisticians made A LOT of calculations by hand to come up with the "best explanation" at the time?

Just how did statisticians come up with this formula in the early 19th century with the aid of computers? Manual or there is more to it than meets the eye?

$\endgroup$
40
$\begingroup$

The correction is called Bessel's correction and it has a mathematical proof. Personally, I was taught it the easy way: using $n-1$ is how you correct the bias of $E[\frac{1}{n}\sum_1^n(x_i - \bar x)^2]$ (see here).

You can also explain the correction based on the concept of degrees of freedom, simulation isn't strictly needed.

$\endgroup$
  • 16
    $\begingroup$ Proof alternate #3 has a beautiful intuitive explanation that even a lay person can understand. The basic idea is that the sample mean is not the same as the population mean. Your observations are naturally going to be closer to the sample mean than the population mean, and this ends up underestimating those $(x_i - \mu)^2$ terms with $(x_i - \bar{x})^2$ terms. This is probably obvious to most people but I never thought about the "intuition" as to why the biased sample variance is biased until now. I only learned the formal proofs. $\endgroup$ – WetlabStudent May 26 '14 at 4:52
  • 2
    $\begingroup$ There is also a geometrical approach why to correct with n-1 (explained very nicely in Saville and Wood: Statistical Methods: The Geometric Approach). Put it shortly: A sample of n can be regarded as an n-dimensional data space. The sample point vectors add to a observed vector which can be decomposed to a model vector with p-dimension corresponding to p parameter and an error vector with n-p dimension. The corresponding Pythagorean breakup of the error vector has n-p squares which average is a measure for the variation. $\endgroup$ – giordano May 27 '14 at 16:42
  • $\begingroup$ I will give you a beautiful link which contains a brief explanation: en.wikipedia.org/wiki/Bias_of_an_estimator $\endgroup$ – Christina Jan 29 '15 at 21:50
  • $\begingroup$ Can you explain why in the proof (alternate 3) we assume that both true and biased variances calculated using $n$ $x$'s? The problem of different variances arises when we have a population (with true variance) and a sample (with biased variance). But if we calculate variance on the same data, namely $x_1, x_2, ... , x_n$, why should they ever differ? There we think of $\sigma^2$ as a true variance calculated using exactly same $x$'s as the biased one $s_\text{biased}^2$. I can't agree with this proof. Please help, what am I missing? $\endgroup$ – Turkhan Badalov Dec 30 '17 at 15:33
56
$\begingroup$

Most proofs I have seen are simple enough that Gauss (however he did it) probably found it pretty easy to prove.

I've been looking for a derivation on CV that I could link you to (there are a number of links to proofs off-site, including at least one in answers here), but I haven't found one here on CV in a couple of searches, so for the sake of completeness, I'll give a simple one. Given its simplicity, it's easy to see how people would start to use what's usually called Bessel's correction.

This takes $E(X^2)=\text{Var}(X) + E(X)^2$ as assumed knowledge, and assumes that the first few basic variance properties are known.

\begin{eqnarray} E[\sum_{i=1}^{n} (x_i-\bar x)^2] &=& E[\sum_{i=1}^{n} x_i^2-2\bar x\sum_{i=1}^{n} x_i+n\bar{x}^2]\\ &=& E[\sum_{i=1}^{n} x_i^2-n\bar{x}^2] \\ &=& n E[x_i^2]- n E[\bar{x}^2]\\ &=& n (\mu^2 + \sigma^2) - n(\mu^2+\sigma^2/n)\\ &=& (n-1) \sigma^2 \end{eqnarray}

$\endgroup$
  • 1
    $\begingroup$ which property makes the term $-2\bar{x}\sum_{i=1}^{n}x_i$ disappear? $\endgroup$ – Ciprian Tomoiagă Sep 28 '16 at 9:01
  • 3
    $\begingroup$ It doesn't disappear. Did you notice the sign of the last term changed? $\endgroup$ – Glen_b -Reinstate Monica Sep 28 '16 at 10:16
  • 2
    $\begingroup$ (+1) I have recently heard a great proof that I personally find more intuitive. Sample variance with with $1/n$ factor can be re-expressed as the average of all squared differences between all pairs points. Now notice that the pairs where the same point enters twice are all zero, and this biases the expression. It seems reasonable to correct the bias by excluding all these pairs from the double sum and only averaging across the rest. This yields Bessel's correction. $\endgroup$ – amoeba says Reinstate Monica Oct 12 '17 at 18:06
  • 1
    $\begingroup$ Nope, never mind, figured it out. $V[\bar{x}] = \frac{V[x]}{n}$, so you're just applying that same identity you mentioned above to both terms in line 3. $\endgroup$ – tel Mar 29 '18 at 8:27
  • 1
    $\begingroup$ Any of the iid variates have the same second moment. We go from talking about all of them to just discussing one of them. You could as easily have taken $x_1$ (and some people do) or $x_2$ or $x_n$... but I have taken the $i$-th $\endgroup$ – Glen_b -Reinstate Monica Mar 29 '18 at 8:34
37
+100
$\begingroup$

According to Weisstein's World of Mathematics, it was first proved by Gauss in 1823. The reference is volume 4 of Gauss' Werke, which can be read at https://archive.org/details/werkecarlf04gausrich. The relevant pages seem to be 47-49. It seems that Gauss investigated the question and came up with a proof. I don't read Latin, but there is a German summary in the text. Pages 103-104 explain what he did (Edit: I added a rough translation):

Allein da man nicht berechtigt ist, die sichersten Werthe fuer die wahren Werthe selbst zu halten, so ueberzeugt man sich leicht, dass man durch dieses Verfahren allemal den wahrscheinlichsten und mittleren Fehler zu klein finden muss, und daher die gegebenen Resultaten eine groessere Genauigkeit beilegt, als sie wirklich besitzen. [But since one is not entitled to treat the most probable values as though they were the actual values, one can easily convince oneself that one must always find that the most probable error and the average error are too small, and that therefore the given results possess a greater accuracy than they really have.]

from which it would seem that it was well-known that the sample variance is a biased estimate of the population variance. The article goes on to say that the difference between the two is usually ignored because it's not important if the sample size is big enough. Then it says:

Der Verfasser hat daher diesen Gegenstand eine besondere Untersuchung unterworfen, die zu einem sehr Merkwuerdigen hoechst einfachen Resultate gefuehrt hat. Man braucht nemlich den nach dem angezeigten fahlerhaften Verfahren gefundenen mittleren Fehler, um ihn in die richtigen zu verwandeln, nur mit

$$\sqrt{\frac{\pi-\rho}{\pi}}$$

zu multiplicieren, wo $\pi$ die Anzahl der beobachtungen (number of observations) und $\rho$ die Anzahl der unbekannten groessen (number of unknowns) bedeutet. [The author has therefore made a special study of this object which has led to a very strange and extremely simple result. Namely, one needs only to multiply the average error found by the above erroneous process by (the given expression) to change it into the right one, where $\pi$ is the number of observations and $\rho$ is the number of unknown quantities.]

So if this is indeed the first time that the correction was found, then it seems that it was found by a clever calculation by Gauss, but people were already aware that some correction was required, so perhaps someone else could have found it empirically before this. Or possibly previous authors didn't care to derive the precise answer because they were working with fairly large data sets anyway.

Summary: manual, but people already knew that $n$ in the denominator wasn't quite right.

$\endgroup$
  • $\begingroup$ If someone could provide a translation of the German, that would be nice. I for one don't read German. $\endgroup$ – Faheem Mitha Jun 10 '14 at 23:52
  • 2
    $\begingroup$ Yes, Google Translate doesn't work so well because of my spelling errors! I'll add in an attempt at translation; it will be a good way of practising my German. $\endgroup$ – Flounderer Jun 10 '14 at 23:55
14
$\begingroup$

For me one piece of intuition is that

$$\begin{array}{c} \mbox{The degree to which}\\ X_{i}\mbox{ varies from }\bar{X} \end{array}+\begin{array}{c} \mbox{The degree to which}\\ \bar{X}\mbox{ varies from }\mu \end{array}=\begin{array}{c} \mbox{The degree to which }\\ X_{i}\mbox{ varies from }\mu. \end{array}$$

That is,

$$\mathbf{E}\left[\left(X_{i}-\bar{X}\right)^{2}\right]+\mathbf{E}\left[\left(\bar{X}-\mu\right)^{2}\right]=\mathbf{E}\left[\left(X_{i}-\mu\right)^{2}\right].$$

Actually proving the above equation takes a bit of algebra (this algebra is very similar to @Glen_b's answer above). But assuming it is true, we can rearrange to get:

$$\mathbf{E}\left[\left(X_{i}-\bar{X}\right)^{2}\right]=\underset{\sigma^{2}}{\underbrace{\mathbf{E}\left[\left(X_{i}-\mu\right)^{2}\right]}}-\underset{\frac{\sigma^{2}}{n}}{\underbrace{\mathbf{E}\left[\left(\bar{X}-\mu\right)^{2}\right]}}=\frac{n-1}{n}\sigma^2.$$

For me, another piece of intuition is that using $\bar{X}$ instead of $\mu$ introduces bias. And this bias is exactly equal to $\mathbf{E}\left[\left(\bar{X}-\mu\right)^{2}\right]=\frac{\sigma^2}{n}$.

$\endgroup$
12
$\begingroup$

Most of the answers have already elaborately explained it but apart from those there's one simple illustration that one could find helpful:

Suppose you are given that $n=4$ and the first three numbers are:

$8,4,6$,_

Now the fourth number can be anything since there are no constraints. Now consider the situation when you are given that $n=4$ and $\bar x=6$, then if the first three numbers are: $8,4,6$ then the fourth number has to be $6$.

This is to say that if you know $n-1$ values and $\bar x$, then the $nth$ value has no freedom. Thus $n-1$ gives us an unbiased estimator.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.