248
$\begingroup$

I was asked today in class why you divide the sum of square error by $n-1$ instead of with $n$, when calculating the standard deviation.

I said I am not going to answer it in class (since I didn't wanna go into unbiased estimators), but later I wondered - is there an intuitive explanation for this?!

$\endgroup$
6
  • 48
    $\begingroup$ I'd like to quote this zinger from the book Numerical Recipes: "...if the difference between $n$ and $n-1$ ever matters to you, then you are probably up to no good anyway - e.g., trying to substantiate a questionable hypothesis with marginal data." $\endgroup$ Commented Oct 25, 2010 at 14:09
  • 18
    $\begingroup$ a really elegant, intuitive explanation is presented here (below the proof) en.wikipedia.org/wiki/… The basic idea is that your observations are, naturally, going to be closer to the sample mean than the population mean. $\endgroup$ Commented May 26, 2014 at 16:23
  • 13
    $\begingroup$ @Tal, This is why schools suck. You ask them "why this?", and they reply "just memorize it". $\endgroup$
    – Pacerier
    Commented Jun 3, 2015 at 11:51
  • 5
    $\begingroup$ If you are looking for an intuitive explanation, you should see the reason for yourself by actually taking samples! Watch this, it precisely answers you question. youtube.com/watch?v=xslIhnquFoE $\endgroup$ Commented Sep 24, 2015 at 23:36
  • 2
    $\begingroup$ But why n - 1 and not n - 2 or n - 0.5? $\endgroup$
    – maalls
    Commented Jul 19, 2022 at 7:03

20 Answers 20

157
$\begingroup$

The standard deviation calculated with a divisor of $n-1$ is a standard deviation calculated from the sample as an estimate of the standard deviation of the population from which the sample was drawn. Because the observed values fall, on average, closer to the sample mean than to the population mean, the standard deviation which is calculated using deviations from the sample mean underestimates the desired standard deviation of the population. Using $n-1$ instead of $n$ as the divisor corrects for that by making the result a little bit bigger.

Note that the correction has a larger proportional effect when $n$ is small than when it is large, which is what we want because when n is larger the sample mean is likely to be a good estimator of the population mean.

When the sample is the whole population we use the standard deviation with $n$ as the divisor because the sample mean is population mean.

(I note parenthetically that nothing that starts with "second moment recentered around a known, definite mean" is going to fulfil the questioner's request for an intuitive explanation.)

$\endgroup$
8
  • 27
    $\begingroup$ Let's not confuse "intuitive" with "nontechnical". $\endgroup$
    – whuber
    Commented Oct 24, 2010 at 15:38
  • 71
    $\begingroup$ @Michael, This doesn't explain Why do we use n−1 instead of n−2 (or even n−3)? $\endgroup$
    – Pacerier
    Commented Jun 3, 2015 at 11:52
  • 3
    $\begingroup$ @Pacerier Have a look at Whuber's answer below for detail on that point. In essence, the correction is n-1 rather than n-2 etc because the n-1 correction gives results that are very close to what we need. More exact corrections are shown here: en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation $\endgroup$ Commented Jun 3, 2015 at 21:37
  • 2
    $\begingroup$ Hi @Michael, so why deviation calculated from sample mean tends to be smaller than population mean? $\endgroup$
    – Allen Qin
    Commented Nov 15, 2016 at 3:57
  • 3
    $\begingroup$ "Because the observed values fall, on average, closer to the sample mean than to the population mean, the standard deviation which is calculated using deviations from the sample mean underestimates the desired standard deviation of the population." Why the sample mean always underestimates? What if it overestimates? $\endgroup$ Commented Nov 20, 2016 at 8:03
146
$\begingroup$

Variance is simply defined as the average of the squared deviations from the mean. Accordingly, we should calculate the variance using the following formulas:

Variance of the population:

$\sigma^2= \frac{\sum_{i}^{N}(X_i-\mu)^2}{N}$ where $\mu$ is the mean and $N$ is the size of the population.

Variance of a sample (e.g. sample $t$):

$\sigma^2_t= \frac{\sum_{i}^{n}(X_i-\overline{X})^2}{n}$ where $\overline{X}$ is the mean and $n$ is the size of this small sample.

However, we select a sample not just to describe a small part, but to make inferences about the whole population. We try to estimate the population variance only by using the values from the sample. For inference, we define sample variance as an estimator of the population variance. The aim and the formula of the good sample variance $S^2$ is different from those of the useless variance of a sample $\sigma^2_t$. We will try to derive a formula for sample variance.

We have two random variables, the trait $X$ and the sample mean $\overline{X}$. Sample mean is a random variable because it gets different values from sample to sample. The mean and variance of it can be derived easily:

$E(\overline{X}) =E(\frac{1}{n}\sum_{i}^{n}X_i) =\frac{1}{n}E(\sum_{i}^{n}X_i) =\frac{1}{n}\sum_{i}^{n}E(X_i) =\frac{1}{n}n\mu=\mu$

$Var(\overline{X}) =Var(\frac{1}{n}\sum_{i}^{n}X_i) =\frac{1}{n^2}Var(\sum_{i}^{n}X_i) =\frac{1}{n^2}\sum_{i}^{n}Var(X_i) =\frac{1}{n^2}n\sigma^2=\frac{\sigma^2}{n}$

As a trivial note, the deviations of individual observations from the sample mean looks larger than the deviations of sample means from the population mean (i.e. $\sigma^2_t > \frac{\sigma^2}{n}$). It makes sense because taking average smooths extreme values of individual observations.

To make inference about the population variance, note that the deviatons of $X$ from $\mu$ involves two kinds of deviations. First, within a sample, the random variable $X$ deviates from sample mean $\overline{X}$ with variance $\sigma^2_t$. Second, between samples, the random variable $\overline{X}$ also deviates from $\mu$ with variance $\frac{\sigma^2}{n}$. Intiutively, we can add up within sample and between samples variances and get $\sigma^2=\sigma^2_t+\frac{\sigma^2}{n}$.

After solving for the population variance, we get $\sigma^2=\sigma^2_t \times\frac{n}{n-1}$. Note that, this is not a formal definition of population variance. Although we can't calculate the population variance using sample observations, we just derived the formula for its estimator, the so-called sample variance.

$S^2=\sigma^2_t \times\frac{n}{n-1} =\frac{\sum_{i}^{n}(X_i-\overline{X})^2}{n-1}$.

If adding up two kind of variances doesn't make sense, one can also prove that $E[S^2]=\sigma^2$ is true.

Most of the confusion about the division by $n-1$ comes from the misconception that the sample variance $S^2$ is the same thing as the variance of sample $\sigma^2_t$. This answer tries to distinguish them and give intiutive proof for sample variance.

$\endgroup$
2
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ Commented May 18, 2022 at 11:16
  • $\begingroup$ What does "of the a sample" mean? Is that a typo? $\endgroup$
    – Wyck
    Commented Nov 23, 2023 at 14:51
69
$\begingroup$

A common one is that the definition of variance (of a distribution) is the second moment recentered around a known, definite mean, whereas the estimator uses an estimated mean. This loss of a degree of freedom (given the mean, you can reconstitute the dataset with knowledge of just $n-1$ of the data values) requires the use of $n-1$ rather than $n$ to "adjust" the result.

Such an explanation is consistent with the estimated variances in ANOVA and variance components analysis. It's really just a special case.

The need to make some adjustment that inflates the variance can, I think, be made intuitively clear with a valid argument that isn't just ex post facto hand-waving. (I recollect that Student may have made such an argument in his 1908 paper on the t-test.) Why the adjustment to the variance should be exactly a factor of $n/(n-1)$ is harder to justify, especially when you consider that the adjusted SD is not an unbiased estimator. (It is merely the square root of an unbiased estimator of the variance. Being unbiased usually does not survive a nonlinear transformation.) So, in fact, the correct adjustment to the SD to remove its bias is not a factor of $\sqrt{n/(n-1)}$ at all!

Some introductory textbooks don't even bother introducing the adjusted sd: they teach one formula (divide by $n$). I first reacted negatively to that when teaching from such a book but grew to appreciate the wisdom: to focus on the concepts and applications, the authors strip out all inessential mathematical niceties. It turns out that nothing is hurt and nobody is misled.

$\endgroup$
8
  • 2
    $\begingroup$ Thank you Whuber. I have to teach the students with the n-1 correction, so dividing in n alone is not an option. As written before me, to mention the connection to the second moment is not an option. Although to mention how the mean was already estimated thereby leaving us with less "data" for the sd - that's important. Regarding the bias of the sd - I remembered encountering it - thanks for driving that point home. Best, Tal $\endgroup$
    – Tal Galili
    Commented Oct 24, 2010 at 7:15
  • 3
    $\begingroup$ @Tal I was writing in your language, not that of your students, because I am confident you are fully capable of translating it into whatever you know will reach them. In other words, I interpreted "intuitive" in your question to mean intuitive to you. $\endgroup$
    – whuber
    Commented Oct 24, 2010 at 15:40
  • 1
    $\begingroup$ Hi Whuber. Thank you for the vote of confidence :). The loose of the degree of freedom for the estimation of the expectancy is one that I was thinking of using in class. The problem is that the concept of "degrees of freedom" by itself is one that needs knowledge/intuition. But combining it with some of the other answers given in this thread will be useful (to me, and I hope others in the future). Best, Tal $\endgroup$
    – Tal Galili
    Commented Oct 24, 2010 at 21:12
  • $\begingroup$ For large $n$, there isn't typically much difference between dividing by $n$ or $n-1$, so it would be acceptable to introduce the uncorrected formula provided it was intended to apply to large samples, no? $\endgroup$
    – PatrickT
    Commented Sep 16, 2017 at 14:45
  • 1
    $\begingroup$ @Patrick You might be reading too much into my answer, because it is explicit about the reasons: they are pedagogical and have nothing to do with whether $n$ is large or not. $\endgroup$
    – whuber
    Commented Sep 18, 2017 at 12:58
23
$\begingroup$

Why divide by $n-1$ rather than $n$? Because it is customary, and results in an unbiased estimate of the variance. However, it results in a biased (low) estimate of the standard deviation, as can be seen by applying Jensen's inequality to the concave function, square root.

So what's so great about having an unbiased estimator? It does not necessarily minimize mean square error. The MLE for a Normal distribution is to divide by $n$ rather than $n-1$. Teach your students to think, rather than to regurgitate and mindlessly apply antiquated notions from a century ago.

$\endgroup$
5
  • 13
    $\begingroup$ (+1) The more I think about this situation (and I've given it some real thought, to the extent of researching the earlier papers such as Student's 1908 Biometrica contribution to try to track down when and why $n-1$ made its appearance), the more I think that "because it's customary" is the only possible correct answer. I am unhappy to see the downvotes and can only guess that they are responding to the last sentence, which could easily be seen as attacking the O.P., even though I doubt that was your intention. $\endgroup$
    – whuber
    Commented Aug 28, 2015 at 17:45
  • 2
    $\begingroup$ My last sentence was friendly advice to all concerned, as opposed to an attack on the OP. $\endgroup$ Commented Aug 29, 2015 at 1:50
  • $\begingroup$ In much use it will not matter, when used in tests or for confidence intervals one would have to adjust other parts of the procedure and in the end obtain the same result! $\endgroup$ Commented Aug 25, 2017 at 16:55
  • 1
    $\begingroup$ (+1) It indeed does not minimise mean square error - if that is the goal one should instead divide by $n+1$. $\endgroup$ Commented Jan 11, 2021 at 14:32
  • $\begingroup$ @whuber I've plotted what happens if we use n-0, n-2 and other values in my answer below to show this visually. $\endgroup$ Commented Oct 13, 2023 at 3:55
23
$\begingroup$

It is well-known (or easily proved) that the quadratic $\alpha z^2 + 2\beta z + \gamma$ has an extremum at $z = -\frac{\beta}{\alpha}$ which point is midway between the roots $\frac{-\beta - \sqrt{\beta^2-\alpha\gamma}}{\alpha}$ and $\frac{-\beta + \sqrt{\beta^2-\alpha\gamma}}{\alpha}$ of the quadratic. This shows that, for any given $n$ real numbers $x_1, x_2, \ldots, x_n$, the quantity $$G(a) = \sum_{i=1}^n (x_i-a)^2 = \left(\sum_{i=1}^n x_i^2\right) -2a\left(\sum_{i=1}^n x_i\right) + na^2,$$ has minimum value when $\displaystyle a = \frac 1n \sum_{i=1}^n x_i =\bar{x}$.

Now, suppose that the $x_i$ are a sample of size $n$ from a distribution with unknown mean $\mu$ and unknown variance $\sigma^2$. We can estimate $\mu$ as $\frac 1n \sum_{i=1}^n x_i = \bar{x}$ which is easy enough to calculate, but an attempt to estimate $\sigma^2$ as $\frac 1n \sum_{i=1}^n (x_i-\mu)^2 = n^{-1}G(\mu)$ encounters the problem that we don't know $\mu$. We can, of course, readily compute $G(\bar{x})$ and we know that $G(\mu) \geq G(\bar{x})$, but how much larger is $G(\mu)$? The answer is that $G(\mu)$ is larger than $G(\bar{x})$ by a factor of approximately $\frac{n}{n-1}$, that is, $$G(\mu) \approx \frac{n}{n-1}G(\bar{x})\tag{1}$$ and so the estimate $\displaystyle n^{-1}G(\mu)= \frac 1n\sum_{i=1}^n(x_i-\mu)^2$ for the variance of the distribution can be approximated by $\displaystyle \frac{1}{n-1}G(\bar{x}) = \frac{1}{n-1}\sum_{i=1}^n (x_i-\bar{x})^2.$

So, what is an intuitive explanation of $(1)$? Well, we have that \begin{align} G(\mu) &= \sum_{i=1}^n (x_i-\mu)^2\\ &= \sum_{i=1}^n (x_i-\bar{x} + \bar{x}-\mu)^2\\ &= \sum_{i=1}^n \left((x_i-\bar{x})^2 + (\bar{x}-\mu)^2 + 2(x_i-\bar{x})(\bar{x}-\mu)\right)\\ &= G(\bar{x}) + n(\bar{x}-\mu)^2 + (\bar{x}-\mu)\sum_{i=1}^n(x_i-\bar{x})\\ &= G(\bar{x}) + n(\bar{x}-\mu)^2 \tag{2} \end{align} since $\sum_{i=1}^n (x_i-\bar{x}) = n\bar{x}-n\bar{x} = 0$. Now, \begin{align} n(\bar{x}-\mu)^2 &= n\frac{1}{n^2}\left(\sum_{i=1}^n(x_i-\mu)\right)^2\\ &= \frac 1n \sum_{i=1}^n(x_i-\mu)^2 + \frac 2n \sum_{i=1}^n\sum_{j=i+1}^n(x_i-\mu)(x_j-\mu)\\ &= \frac 1n G(\mu) + \frac 2n \sum_{i=1}^n\sum_{j=i+1}^n(x_i-\mu)(x_j-\mu)\tag{3} \end{align} Except when we have an extraordinarily unusual sample in which all the $x_i$ are larger than $\mu$ (or they are all smaller than $\mu$), the summands $(x_i-\mu)(x_j-\mu)$ in the double sum on the right side of $(3)$ take on positive as well as negative values and thus a lot of cancellations occur. Thus, the double sum can be expected to have small absolute value, and we simply ignore it in comparison to the $\frac 1nG(\mu)$ term on the right side of $(3)$. Thus, $(2)$ becomes $$G(\mu) \approx G(\bar{x}) + \frac 1nG(\mu) \Longrightarrow G(\mu) \approx \frac{n}{n-1}G(\bar{x})$$ as claimed in $(1)$.

$\endgroup$
2
  • 29
    $\begingroup$ Only on this stack exchange would this ever be considered an intuitive answer. $\endgroup$ Commented Mar 13, 2017 at 3:15
  • 2
    $\begingroup$ Strangely, this is indeed intuitive $\endgroup$ Commented Jan 16, 2023 at 9:39
20
$\begingroup$

This is a total intuition, but the simplest answer is that is a correction made to make standard deviation of one-element sample undefined rather than 0.

$\endgroup$
5
  • 14
    $\begingroup$ Why not, then, use $\frac{n}{n^2-1}$ or even $\frac{1}{\exp(1)-\exp(1/n)}$ as corrections? :-) $\endgroup$
    – whuber
    Commented Sep 22, 2011 at 6:25
  • 2
    $\begingroup$ @whuber Parsimony (-; $\endgroup$
    – user88
    Commented Sep 22, 2011 at 10:17
  • 4
    $\begingroup$ $\frac{1}{n-1}$ is even more "parsimonious". :-) $\endgroup$
    – whuber
    Commented Sep 22, 2011 at 15:02
  • 3
    $\begingroup$ @mbq, Regarding your answer ~"it's a correction made to make standard deviation of one-element sample undefined rather than 0", is that really the reason why, or is this a joke answer? You know non-mathers like us can't tell. $\endgroup$
    – Pacerier
    Commented Jun 3, 2015 at 11:56
  • 5
    $\begingroup$ Formally, it is a consequence than reason, but, as i wrote, I find it to be a good intution to memorize it. $\endgroup$
    – user88
    Commented Jun 3, 2015 at 17:45
19
$\begingroup$

At the suggestion of whuber, this answer has been copied over from another similar question.

Bessel's correction is adopted to correct for bias in using the sample variance as an estimator of the true variance. The bias in the uncorrected statistic occurs because the sample mean is closer to the middle of the observations than the true mean, and so the squared deviations around the sample mean systematically underestimates the squared deviations around the true mean.

To see this phenomenon algebraically, just derive the expected value of a sample variance without Bessel's correction and see what it looks like. Letting $S_*^2$ denote the uncorrected sample variance (using $n$ as the denominator) we have:

$$\begin{equation} \begin{aligned} S_*^2 &= \frac{1}{n} \sum_{i=1}^n (X_i - \bar{X})^2 \\[8pt] &= \frac{1}{n} \sum_{i=1}^n (X_i^2 - 2 \bar{X} X_i + \bar{X}^2) \\[8pt] &= \frac{1}{n} \Bigg( \sum_{i=1}^n X_i^2 - 2 \bar{X} \sum_{i=1}^n X_i + n \bar{X}^2 \Bigg) \\[8pt] &= \frac{1}{n} \Bigg( \sum_{i=1}^n X_i^2 - 2 n \bar{X}^2 + n \bar{X}^2 \Bigg) \\[8pt] &= \frac{1}{n} \Bigg( \sum_{i=1}^n X_i^2 - n \bar{X}^2 \Bigg) \\[8pt] &= \frac{1}{n} \sum_{i=1}^n X_i^2 - \bar{X}^2. \end{aligned} \end{equation}$$

Taking expectations yields:

$$\begin{equation} \begin{aligned} \mathbb{E}(S_*^2) &= \frac{1}{n} \sum_{i=1}^n \mathbb{E}(X_i^2) - \mathbb{E} (\bar{X}^2) \\[8pt] &= \frac{1}{n} \sum_{i=1}^n (\mu^2 + \sigma^2) - (\mu^2 + \frac{\sigma^2}{n}) \\[8pt] &= (\mu^2 + \sigma^2) - (\mu^2 + \frac{\sigma^2}{n}) \\[8pt] &= \sigma^2 - \frac{\sigma^2}{n} \\[8pt] &= \frac{n-1}{n} \cdot \sigma^2 \\[8pt] \end{aligned} \end{equation}$$

So you can see that the uncorrected sample variance statistic underestimates the true variance $\sigma^2$. Bessel's correction replaces the denominator with $n-1$ which yields an unbiased estimator. In regression analysis this is extended to the more general case where the estimated mean is a linear function of multiple predictors, and in this latter case, the denominator is reduced further, for the lower number of degrees-of-freedom.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the proof! $\endgroup$
    – upupming
    Commented Jun 25, 2019 at 3:18
18
$\begingroup$

You can gain a deeper understanding of the $n-1$ term through geometry alone, not just why it's not $n$ but why it takes exactly this form, but you may first need to build up your intuition cope with $n$-dimensional geometry. From there, however, it's a small step to a deeper understanding of degrees of freedom in linear models (i.e. model df & residual df). I think there's little doubt that Fisher thought this way. Here's a book that builds it up gradually:

Saville DJ, Wood GR. Statistical methods: the geometric approach. 3rd edition. New York: Springer-Verlag; 1991. 560 pages. 9780387975177

(Yes, 560 pages. I did say gradually.)

$\endgroup$
3
  • 1
    $\begingroup$ Thanks onestop - I didn't think there would be an answer from that direction. Any way to sum-up the intuition, or is that not likely to be possible? Cheers, Tal $\endgroup$
    – Tal Galili
    Commented Oct 24, 2010 at 11:20
  • $\begingroup$ I couldn't do so myself, but a book reviewer summarised the approach in a paragraph in Amer. Stat. in 1993: jstor.org/stable/2684984. I'm not sure it's really practical to use this approach with your students unless you adopt it for the entire course though. $\endgroup$
    – onestop
    Commented Oct 24, 2010 at 12:10
  • 4
    $\begingroup$ Can you summarise a bit of the intuition rather than just a book reference? $\endgroup$
    – oliversm
    Commented Sep 30, 2019 at 18:34
14
$\begingroup$

The estimator of the population variance is biased when applied on a sample of the population. In order to adjust for that bias on needs to divide by n-1 instead of n. One can show mathematically that the estimator of the sample variance is unbiased when we divide by n-1 instead of n. A formal proof is provided here:

https://economictheoryblog.com/2012/06/28/latexlatexs2/

Initially it was the mathematical correctness that led to the formula, I suppose. However, if one wants to add intuition to a formula the already mentioned suggestions appear reasonable.

First, observations of a sample are on average closer to the sample mean than to the population mean. The variance estimator makes use of the sample mean and as a consequence underestimates the true variance of the population. Dividing by n-1 instead of n corrects for that bias.

Furthermore, dividing by n-1 make the variance of a one-element sample undefined rather than zero.

$\endgroup$
8
$\begingroup$

Sample variance can be thought of to be the exact mean of the pairwise "energy" $(x_i-x_j)^2/2$ between all sample points. The definition of sample variance then becomes $$ s^2 = \frac{2}{n(n-1)}\sum_{i< j}\frac{(x_i-x_j)^2}{2} = \frac{1}{n-1}\sum_{i=1}^n(x_i-\bar{x})^2 .$$

This also agrees with defining variance of a random variable as the expectation of the pairwise energy, i.e. let $X$ and $Y$ be independent random variables with the same distribution, then $$ V(X) = E\left(\frac{(X-Y)^2}{2}\right) = E((X-E(X))^2) . $$

To go from the random variable defintion of variance to the defintion of sample variance is a matter of estimating a expectation by a mean which is can be justified by the philosophical principle of typicality: The sample is a typical representation the distribution. (Note, this is related to, but not the same as estimation by moments.)

$\endgroup$
8
  • 2
    $\begingroup$ I couldn't quite follow you at the last paragraph. Isn't mathematical fact that $ V(X) = E\left(\frac{(X-Y)^2}{2}\right) = E((X-E(X))^2)$? Even though the equation is interesting, I don't get how it could be used to teach n-1 intuitively? $\endgroup$
    – KH Kim
    Commented Jun 17, 2012 at 1:59
  • 4
    $\begingroup$ I like this approach, but it omits a key idea: to compute the mean energy between all pairs of sample points, one would have to count the values $(x_i-x_i)^2$, even though they are all zero. Thus the numerator of $s^2$ remains the same but the denominator ought to be $n$, not $n-1$. This shows the sleight-of-hand that has occurred: somehow, you need to justify not including such self-pairs. (Because they are included in the analogous population definition of variance, this is not an obvious thing.) $\endgroup$
    – whuber
    Commented Jun 24, 2015 at 17:24
  • $\begingroup$ @whuber To complete this answer's argument in an intuitively elegant way: the self-pairs indeed are in the analogous population definition of variance as well, but their proportion tends to 0 as the number of possible pairs you could draw goes to infinity. $\endgroup$
    – zwets
    Commented Jul 20, 2020 at 10:56
  • $\begingroup$ So to complete the answer I'd argue the opposite of @whuber final point: self-pairs must be excluded because they are not in the population variance either. Their proportion ${n \over {n^2}} = {1 \over n} \to 0$ as $n \to \inf$. The $n \over {n-1}$ factor corrects precisely for this over-representation of self-pairs in the sample vs the population. Of the $n^2$ pairs in a sample, $n$ are self-pairs, whereas $n^2 - n$ follow population variance. Hence we multiply $s^2$ by ${n^2 \over {n^2 - n}} = {n \over {n - 1}}$ to remove the self-pairs from the denominator. $\endgroup$
    – zwets
    Commented Jul 20, 2020 at 13:32
  • $\begingroup$ @zwets I find your argument interesting but less than useful because of two flaws: You assume an infinite population (and such an assumption is both superfluous and overly restrictive) and you appear to confound the population size with the sample size, referring to both as $n.$ $\endgroup$
    – whuber
    Commented Jul 20, 2020 at 14:02
6
$\begingroup$

The sample mean is defined as $\bar{X} = \frac{1}{n}\sum_{i=1}^{n} X_i$, which is quite intuitive. But the sample variance is $S^2 = \frac{1}{n-1}\sum_{i=1}^{n} (X_i - \bar{X})^2$. Where did the $n - 1$ come from ?

To answer this question, we must go back to the definition of an unbiased estimator. An unbiased estimator is one whose expectation tends to the true expectation. The sample mean is an unbiased estimator. To see why:

$$ E[\bar{X}] = \frac{1}{n}\sum_{i=1}^{n} E[X_i] = \frac{n}{n} \mu = \mu $$

Let us look at the expectation of the sample variance,

$$ S^2 = \frac{1}{n-1} \sum_{i=1}^{n} (X_i^2) - n\bar{X}^2 $$

$$ E[S^2] = \frac{1}{n-1} \left( n E[(X_i^2)] - nE[\bar{X}^2] \right). $$

Notice that $\bar{X}$ is a random variable and not a constant, so the expectation $E[\bar{X}^2] $ plays a role. This is the reason behind the $n-1$.

$$E[S^2] = \frac{1}{n-1} \left( n (\mu^2 + \sigma^2) - n(\mu^2 + Var(\bar{X})) \right). $$ $$Var(\bar{X}) = Var(\frac{1}{n}\sum_{i=1}^{n} X_i) = \sum_{i=1}^{n} \frac{1}{n^2} Var(X_i) = \frac{\sigma^2}{n} $$

$$E[S^2] = \frac{1}{n-1} \left( n (\mu^2 + \sigma^2) - n(\mu^2 + \sigma^2/n) \right). = \frac{(n-1)\sigma^2}{n-1} = \sigma^2 \\ $$

As you can see, if we had the denominator as $n$ instead of $n-1$, we would get a biased estimate for the variance! But with $n-1$ the estimator $S^2$ is an unbiased estimator.

$\endgroup$
1
  • 4
    $\begingroup$ But it doesn't follow that $S$ is an unbiased estimator of the standard deviation. $\endgroup$ Commented Jan 15, 2016 at 17:32
5
$\begingroup$

Suppose that you have a random phenomenon. Suppose again that you only get one $N=1$ sample, or realization, $x$. Without further assumptions, your "only" reasonable choice for a sample average is $\overline{m}=x$. If you do not subtract $1$ from your denominator, the (uncorrect) sample variance would be $$ V=\frac{\sum_N (x_n - \overline{m} )^2}{N}$$, or:

$$\overline{V}=\frac{(x-\overline{m})^2}{1} = 0\,.$$

Oddly, the variance would be null with only one sample. And having a second sample $y$ would risk to increase your variance, if $x\neq y$. This makes no sense. Intuitively, an infinite variance would be a sounder result, and you can recovered it only by "dividing by $N-1=0$".

Estimating a mean is fitting a polynomial with degree $0$ to the data, having one degree of freedom (dof). This Bessel's correction applies to higher degrees of freedom models too: of course you can fit perfectly $d+1$ points with a $d$ degree polynomial, with $d+1$ dofs. The illusion of a zero-squared-error can only be counterbalanced by dividing by the number of points minus the number of dofs. This issue is particularly sensitive when dealing with very small experimental datasets.

$\endgroup$
2
  • $\begingroup$ It is unclear why "an infinite variance would be a sounder result" than a zero variance. Indeed, you seem to use "sample variance" in the sense of a variance estimator, which is more confusing yet. $\endgroup$
    – whuber
    Commented Jun 26, 2017 at 16:42
  • 1
    $\begingroup$ I understand. To answer an intuitive explanation between two options, I tried to suggest that one of the two is somehow unacceptable, based on the mundane rule that $0 < \infty$. A rephrasing is indeed necessary, and upcoming $\endgroup$ Commented Jun 26, 2017 at 18:33
3
$\begingroup$

The intuitive reason for the $n-1$ is that the $n$ deviations in the calculation of the standard deviation are not independent. There is one constraint which is that the sum of the deviations is zero. When we take that into account we are effectively dealing with $n-1$ quantities rather than $n$. (Geometrically the deviation vector $x-\bar{x}$ is the projection of $x$ onto the space orthogonal to the space spanned by the vector of all ones and the space onto which it projects has dimension $n-1$.)

$\endgroup$
3
$\begingroup$

To answer the comment by @Pacerier

This doesn't explain Why do we use n−1 instead of n−2 (or even n−3)?

We can do modeling to simulate what happens if we use n-3, n-2, n-1, n-0, n+1, n+2, n+3, compared to the true Standard Deviation.

enter image description here

n-0 has a higher absolute error than n-1. n-2 has a similar absolute error but tends to over-estimate the standard deviation rather than under-estimating it like n-1 does. We can also plot n-1, n-1.25, n-1.5 and n-1.75 to see if intermediate values would work:

enter image description here

So it seems like n-1.5 yields somewhat more accurate approximations but this only really matters at very small sample sizes.

$\endgroup$
1
$\begingroup$

Generally using "n" in the denominator gives smaller values than the population variance which is what we want to estimate. This especially happens if the small samples are taken. In the language of statistics, we say that the sample variance provides a “biased” estimate of the population variance and needs to be made "unbiased".

If you are looking for an intuitive explanation, you should let your students see the reason for themselves by actually taking samples! Watch this, it precisely answers your question.

https://www.youtube.com/watch?v=xslIhnquFoE

$\endgroup$
0
1
$\begingroup$

We can understand Bessel's correction geometrically by moving into $n$-dimensional space!

We will start with 3 dimensions and then generalize.

Say we have 3 observations $x_1, x_2, x_3$ of a $\mathcal{N}(\mu, \sigma^2)$ random variable. We can visualize this as $3$ points on a number line or a single vector in $\vec{x} \in \mathbb{R}^3$. A cool observation is that the mean $\bar{x}$ can be obtained geometrically by orthogonally projecting $\vec{x}$ in the direction of $\vec{1} = (1, 1,1)^\top$. In other words "The mean is the number $\bar{x}$ for which the vector $(\bar{x},\bar{x},\bar{x})^\top$ is geometrically closest to $(x_1,x_2,x_3)^\top$".

Mean via projection

Say I generate $3$ observations $100$ times. I can visualize this as a cloud of $100$ points in $\mathbb{R}^3$. This cloud will be roughly spherical and centered at $(\mu,\mu,\mu)^\perp$. In fact, each $\vec{x}$ can be thought of as an observation from a multivariate normal $\mathcal{N}(\mu\vec{1}, \sigma^2 I_3)$.

This gives us $100$ different estimates for $\mu$. We can see this below: the $100$ yellow dots each represent a different set of $3$ observations $(x_1,x_2,x_3)$. The blue dots are their projections in the direction of $\vec{1}$ and so represent different estimates of $\mu$. The actual location of $(\mu,\mu,\mu)^\perp$ is marked with the black plus sign. It should be believable that these projections give us an unbiased estimate of $\mu$: the black plus sign should be at the center of the blue dots. So the population mean $\bar{x}$ is an unbiased estimator of the population mean $\mu$.

enter image description here

If I am only given three observations $(x_1,x_2, x_3)$ then the best I can do is estimate $\mu$ with $\bar{x}$. However I do not actually know what $\mu$ is! So I cannot use the projection of the single vector $\vec{x}$ to estimate the variance $\sigma^2$: the variance is defined as the sum of the squared deviations from $\mu$ which I do not know.

So how are we to estimate $\sigma^2$?

This is where things get tricky!

While I don't know $\mu$, I do know that $\mu\vec{1}$ and $\bar{x}\vec{1}$ are both on the same line through the origin. So if I take my orange point cloud and project it orthogonally onto $U = \textrm{span}(\vec{1})^\perp$ we know that the mean of this new blue point cloud really is $\vec{0}$! We now have a dataset with known expected value so we can actually estimate the variance!

This is a 2-dimensional collection of points drawn from a multivariate gaussian with mean $\vec{0}$ and covariance matrix $\sigma^2 I_U$.

Projection onto 1-perp

If $\vec{y} \in \mathbb{R}^k$ is an observation from $\mathcal{N}(\vec{0}, \sigma^2 I_k)$, then $\frac{1}{k} |\vec{y}|^2$ is an unbiased estimate of $\sigma^2$. This is an easy lemma:

$$ \begin{align*} \mathbb{E}\left( \frac{1}{k} |\vec{y}|^2 \right) &= \frac{1}{k} \mathbb{E} \left( \sum_1^k y_i^2\right) \\ &= \frac{1}{k} \sum_1^k \mathbb{E} \left(y_i^2\right) \\ &= \frac{1}{k} (k \sigma^2)\\ &= \sigma^2 \end{align*} $$

So, returning to our blue point cloud $\vec{x} - \bar{x}\vec{1}$ which lives on the $2$ dimensional set $U$ we get that an unbiased estimate of $\sigma^2$ would be

$$\frac{1}{2}|\vec{x} - \bar{x}\vec{1}|^2 = \frac{(x_1 - \bar{x})^2 + (x_2 - \bar{x})^2 + (x_3 - \bar{x})^2}{2}$$

We have obtained Bessel's correction!

How does this generalize to $n$ dimensions?

Exactly the same way!

Now the projection of $\vec{x} - \bar{x}\vec{1}$ lives on the $(n-1)$-dimensional subspace $\operatorname{span}(\vec{1})^\perp$. So an unbiased estimate of $\sigma^2$ is

$$ \frac{1}{n-1}|\vec{x} - \bar{x}\vec{1}|^2 = \frac{1}{n-1} \sum_1^n (x_i - \bar{x})^2 $$

This story is not exactly spelled out in the book "Statistical methods: the Geometric Approach" by David J. Saville and Graham R. Wood but it did inspire this post.

$\endgroup$
1
  • $\begingroup$ I am not entirely pleased with my notation and terminology in this answer when it comes to distributions on subspaces. If anyone has suggestions for improvements I would be grateful. Here is a link to a focused question on this stats.stackexchange.com/q/639210/97124 $\endgroup$ Commented Feb 14 at 23:08
0
$\begingroup$

I think it's worth pointing out the connection to Bayesian estimation. Suppose you assume your data is Gaussian, and so you measure the mean $\mu$ and variance $\sigma^2$ of a sample of $n$ points. You want to draw conclusions about the population. The Bayesian approach would be to evaluate the posterior predictive distribution over the sample, which is a generalized Student's T distribution (the origin of the T-test). This distribution has mean $\mu$, and variance $$\sigma^2 \left(\frac{n+1}{n-1}\right),$$

which is even larger than the typical correction. (It has $2n$ degrees of freedom.)

The generalized Student's T distribution has three parameters and makes use of all three of your statistics. If you decide to throw out some information, you can further approximate your data using a two-parameter normal distribution as described in your question.

From a Bayesian standpoint, you can imagine that uncertainty in the hyperparameters of the model (distributions over the mean and variance) cause the variance of the posterior predictive to be greater than the population variance.

$\endgroup$
0
$\begingroup$

I'm jumping VERY late into this, but would like to offer an answer that is possibly more intuitive than others, albeit incomplete.

As others asserted, the population mean ($\mu$) and the sample mean ($\overline{X}$) are going to differ (where the larger the sample size the smaller the difference).

Let $e$ be the difference (or error) between the population and sample means:

$$ e = \mu - \overline{X} $$

After rearranging:

$$ \overline{X} = \mu - e $$

Thus:

$$ (X_i-\overline{X})^2 = (X_i-(\mu - e))^2 = (X_i - \mu + e)^2 $$

In other words $(X_i-\overline{X})^2$ conceals an error: $(X_i - \mu + e)^2$.

What does this result in?

The table below shows a population of $\{2, 4, 6\}$, so $\mu = 4$, and three possible sample means ($\overline{X}$):

  • $4\ (e = 0)$
  • $3.5\ (e = -0.5)$
  • $4.5\ (e = 0.5)$

The (non-bold) numeric cells shows the squared difference. For example, with $X_1 = 2$ and $\overline{X} = 3.5$, $(X_i-\overline{X})^2 = 2.25$.

The bottom row shows the sum of squares (the numerator in $\frac{\sum(X_i-\overline{X})^2}{n}$), and as you can see, whenever there's an error, it is "overestimated".

To compensate for this, we have to take away something from the denominator.

$$ \begin{array}{|c|c|c|c|} \hline & \overline{X} = \mu = 4 & \overline{X} = 3.5 & \overline{X} = 4.5 \\ \hline X_1 = 2 & 4 & 2.25 & 6.25 \\ \hline X_2 = 4 & 0 & 0.25 & 0.25 \\ \hline X_3 = 6 & 4 & 6.25 & 2.25 \\ \hline \sum(X_i-\overline{X})^2 & \textbf{8} & \textbf{8.75} & \textbf{8.75} \\ \hline \end{array} $$

$\endgroup$
0
$\begingroup$

Here's a very good overview and full proof

In the more general case, note that the sample mean is not the same as the population mean. One's sample observations are naturally going to be closer on average to the sample mean than the population mean, resulting in the average $(x−\bar{x})^2$ value underestimating the average $(x−μ)^2$ value. Thus, $s^2_{biased}$ generally underestimates $σ^2$ with the difference between the two more pronounced when the sample size is small.

$\endgroup$
-3
$\begingroup$

My goodness it's getting complicated! I thought the simple answer was... if you have all the data points you can use "n" but if you have a "sample" then, assuming it's a random sample, you've got more sample points from inside the standard deviation than from outside (the definition of standard deviation). You just don't have enough data outside to ensure you get all the data points you need randomly. The n-1 helps expand toward the "real" standard deviation.

$\endgroup$
1
  • 4
    $\begingroup$ This doesn't make sense. More points from inside the SD than outside? If that means within 1 SD of the mean versus not within, whether that is true has nothing to do with taking a sample. For necessary constraints on fractions within intervals around the mean, see Chebyshev's inequality. To the main question here, "helps expand" doesn't explain $n - 1$ at all, as even granting your argument $n - 2$ might be better still, and so forth, as there is no algebra here, even implicitly. Unfortunately this adds nothing to other answers except a confused set of ideas, either incorrect or irrelevant. $\endgroup$
    – Nick Cox
    Commented Apr 7, 2016 at 1:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.