Timeline for Intuitive explanation for dividing by $n-1$ when calculating standard deviation?
Current License: CC BY-SA 2.5
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 21, 2020 at 9:18 | comment | added | zwets | @whuber I have found a simple proof for finite populations which unfortunately this comment margin is too small to contain. :-) | |
| Jul 20, 2020 at 16:17 | comment | added | whuber♦ | @Zwets Thanks for the clarification. I still don't see how the argument with an infinite population has any bearing on this issue, though. | |
| Jul 20, 2020 at 15:32 | comment | added | zwets | Thanks @whuber, perhaps the limitation of comment space is the issue here. Let me try again: variance of a distribution is half the expected squared distance between arbitrary pairs of values drawn from it. When we estimate this from a sample of size $n$, this includes $n$ self-pairs making up $1/n$ of the pairs. In a population of infinite size, this fraction vanishes, whereas the fraction 'other-pairs', ${n - 1} / n$ tends to 1. I do not confound population and sample size, but indeed do assume population variance applies to an infinite population. | |
| Jul 20, 2020 at 14:02 | comment | added | whuber♦ | @zwets I find your argument interesting but less than useful because of two flaws: You assume an infinite population (and such an assumption is both superfluous and overly restrictive) and you appear to confound the population size with the sample size, referring to both as $n.$ | |
| Jul 20, 2020 at 13:32 | comment | added | zwets | So to complete the answer I'd argue the opposite of @whuber final point: self-pairs must be excluded because they are not in the population variance either. Their proportion ${n \over {n^2}} = {1 \over n} \to 0$ as $n \to \inf$. The $n \over {n-1}$ factor corrects precisely for this over-representation of self-pairs in the sample vs the population. Of the $n^2$ pairs in a sample, $n$ are self-pairs, whereas $n^2 - n$ follow population variance. Hence we multiply $s^2$ by ${n^2 \over {n^2 - n}} = {n \over {n - 1}}$ to remove the self-pairs from the denominator. | |
| Jul 20, 2020 at 10:56 | comment | added | zwets | @whuber To complete this answer's argument in an intuitively elegant way: the self-pairs indeed are in the analogous population definition of variance as well, but their proportion tends to 0 as the number of possible pairs you could draw goes to infinity. | |
| Jun 24, 2015 at 17:24 | comment | added | whuber♦ | I like this approach, but it omits a key idea: to compute the mean energy between all pairs of sample points, one would have to count the values $(x_i-x_i)^2$, even though they are all zero. Thus the numerator of $s^2$ remains the same but the denominator ought to be $n$, not $n-1$. This shows the sleight-of-hand that has occurred: somehow, you need to justify not including such self-pairs. (Because they are included in the analogous population definition of variance, this is not an obvious thing.) | |
| Jun 17, 2012 at 1:59 | comment | added | KH Kim | I couldn't quite follow you at the last paragraph. Isn't mathematical fact that $ V(X) = E\left(\frac{(X-Y)^2}{2}\right) = E((X-E(X))^2)$? Even though the equation is interesting, I don't get how it could be used to teach n-1 intuitively? | |
| Oct 25, 2010 at 9:51 | history | answered | B Student | CC BY-SA 2.5 |