2
$\begingroup$

Recently I've encountered the formula for finite population variance in Davison's and Hinkley's book "Bootstrap Methods and their Application" (see screenshot below). The formula includes $(N - 1)^{-1}$ rather than $N^{-1}$ (as it should be according to wikipedia). I understand that the $(n - 1)^{-1}$ factor appears in the sample variance estimate (i.e. sample taken from the finite population), but here we deal with the entire population.

The source of bias in the sample based estimate is the uncertainty introduced by the mean estimate which by itself is a random variable, but if we're talking about entire population the mean obtained from the population is a fixed quantity and therefore should not effect the variance.

I checked a couple of articles online and I noticed the same definition in some of them as well. For example

Some explanation I found in another question which was posted in 2013: What is the difference between N and N-1 in calculating population variance?. It seems like one reasonable explanation is that using $(N-1)^{-1}$ factor is computationally more convenient (although I don't see any good examples that support the claim) which means that definition of the variance was changed for this particular application since it's no longer the variance in a normal sense, but rather some variation of the discrepancy measurement which gives quite often answers that are very close to the actual variance. The problem is that now the sample variance is biased with respect to the new variance definition, since it's missing $N(N-1)^{-1}$ scaling factor.

I would appreciate if somebody can help me to clarify the points

enter image description here

$\endgroup$
1
  • $\begingroup$ I wonder what your text really means by "var," because with the standard definition (second central moment), the "with replacement" part of (3.15) obviously is incorrect. Consider samples of size $n=1,$ for which the variance of $\bar Y$ reduces to the variance of the population, which requires a denominator of $N,$ not $N-1,$ in its definition. $\endgroup$
    – whuber
    Sep 14 at 16:00

2 Answers 2

1
+200
$\begingroup$

It is better to define the "population variance" with Bessel's correction (i.e., using $N-1$ in the denominator)

You will find that many of the texts in sampling theory give results pertaining to this population quantity, but many will avoid using the term "population variance" to name this quantity. The section of the notes that you highlight in your question is an example of this --- although they state a sampling result in terms of this quantity, they actually avoid giving it a name. Presumably they want to remain agnostic as to what quantity (if any) constitutes the "population variance".

In order to see whether it is better to include or exclude Bessel's correction in the "population variance", let us first examine the properties of the relevant quantities. Suppose you have a finite population of values $Y_1,...,Y_N$ and let $F_N$ denote their empirical distribution. To facilitate our analysis, we will define two alternative population quantities, with and without Bessel's correction:

$$R_N^2 = \frac{1}{N} \sum_{i=1}^N (Y_i-\bar{Y}_N)^2 \quad \quad \quad \quad \quad S_N^2 = \frac{1}{N-1} \sum_{i=1}^N (Y_i-\bar{Y}_N)^2.$$

In the text you are looking at they are giving moments that are (implicitly) conditional on the population values. I will make this conditioning explicit by stating conditioning on $F_N$. If we take a simple-random-sample without replacement from the population (equivalent to assuming that we sample $Y_1,...,Y_n$ from an exchangeable finite population) then it can be shown that:

$$\mathbb{V}(\bar{Y}_n | F_N) = \frac{N-n}{N-1} \cdot \frac{R_N^2}{n} = \frac{N-n}{N} \cdot \frac{S_N^2}{n}.$$

(The latter result is the one that Davison and Hinkley state as $\mathbb{V}(\bar{Y}) = (1-f) n^{-1} \gamma$ in their own notation; I prefer to state it with explicit conditioning and using my own notation.) As you can see from this result, both formulae involve a "finite-population correction" term and a "variance" term. If you use the quantity $R_N^2$ (that does not include Bessel's correction) then you end up shoe-horning Bessel's correction into the finite-population correction term. However, if you use the quantity $S_N^2$ (that does include Bessel's correction) then you get a much cleaner and more natural form for the finite-population correction term.

If we wish to allow a general notion of a "variance" of a set of numbers (as opposed to a variance of a probability distribution), it is natural to ensure that the definition of the "variance" remains consistent as the sample size changes. It is also desirable that it have other useful properties such as being an unbiased estimator with a stable expectation, as is the case for the standard definition of the "sample variance". This is accomplished by defining both the "sample variance" and "population variance" to incorporate Bessel's correction ---i.e., the sample variance is $S_n^2$ and the population variance is $S_N^2$. This makes many results in sampling theory cleaner and simpler, as with the above conditional variance result. It also ensures that the expectation of "the variance" remains fixed and does not "jump down" suddenly at the point where $n=N$. Another desirable property of this approach can be seen if we use model-based sampling theory where we embed the finite population within an infinite superpopulation $Y_1,Y_2,Y_3,...$ with mean $\mu$ and variance $\sigma^2$. In this case both the sample variance $S_n^2$ and the population variance $S_N^2$ is an unbiased estimator of the superpopulation variance $\sigma^2.$

You will find some statisticians who claim that $S_n^2$ is the "sample variance" but that $R_N^2$ should be regarded as the "population variance". This argument is usually made on the basis that $R_N^2$ is the variance of the empirical distribution of the population.$^\dagger$ While that property is indeed true, the argument that this gives a good claim to being the "population variance" is weak and unconvincing --- in particular, it is inconsistent with defining the "sample variance" as $S_n^2$, since the latter is not equal to the variance of the empirical distribution of the sample. This approach also gives rise to several problems, including an immediate and annoying incongruity when we look at "the variance" of a set of numbers. Under this approach, we end up applying Bessel's correction for all $n<N$ but then suddenly the formula changes and the variance "jumps downward" when we get to $n=N$ (as does its expectation, etc.). This introduces all sorts of silliness into the discussion and creates a number of downstream inconsistencies in relevant formulae (e.g., the fact that Bessel's correction ends up getting shoe-horned back into the finite-population correction term). Unfortunately this approach is something that is commonly asserted to bemused students in undergraduate statistics courses, without any convincing argument backing it up. Students naturally wonder why "the variance" of a set of numbers should suddenly jump up at one particular value of $n$ but they are given no satisfying explanation for this (because there is none).

Probably the most damning aspect of treating $S_n^2$ as "the variance" of a sample and $R_N^2$ as "the variance" of the population is that you then literally cannot compute the variance of a set of numbers without knowledge of the population size! This occurs because the relevant formula changes between $n<N$ and $n=N$ so that the variance formula is now a function of $N$ ---i.e., you need to know if the numbers you have are the full population or not. If I give a practitioner holding this position a set of numbers, with no further information, they literally cannot compute "the variance" of those numbers --- extraordinary!

As noted at the start of this answer, many authors in this field sidestep the discussion entirely by refusing to call anything the "population variance" and merely referring to the quantities at issue with their algebraic notation. While this is a defensible position, I prefer to take the approach of saying that there should be a meaningful definition of "the variance" of any set of numbers, and the quantity $S_N^2$ is the one that fits the bill for the population values. You will find that if you adopt this view, a whole lot of things in sampling theory become a lot simpler and cleaner.


$^\dagger$ That is, it can be shown that:

$$R_N^2 = \int \limits_\mathbb{R} (y-\bar{Y}_N)^2 dF_N(y).$$

A secondary argument sometimes presented is that this quantity is equal to the conditional variance of a single sampled value when we condition on the population values ---i.e., we have $R_N^2 = \mathbb{E}(Y_i|F_N)$. That is also not a good basis to define the quantity as the "population variance" since this latter result refers only to variability in a single value.

$\endgroup$
12
  • 1
    $\begingroup$ Regarding the inconsistency with the definition of the sample variance: note that there are authors that define the sample variance (with explicit reference to the moments of the empirical distribution) as $\frac{1}{n} \sum_{i=1}^n (Y_i-\bar{Y}_n)^2$. IMHO this is equally defensible. $\endgroup$
    – statmerkur
    Sep 16 at 14:32
  • $\begingroup$ I find these arguments unconvincing. One of the reasons is that $S_N^2$ is undefined for populations with $N=1.$ This alone ought to override the subjective "unnaturality" of $(N-n)/(N-1)$ compared to $(N-n)/N.$ Another is that the dismissal of the empirical distribution argument appears to be based on an error: the variance of the empirical distribution is $R_N^2.$ A third is that "Bessel's correction" is used for estimators rather than to characterize properties of populations. A fourth is that the with-replacement claim made in the quotation is incorrect. $\endgroup$
    – whuber
    Sep 16 at 21:56
  • $\begingroup$ @whuber: (1) The correction term $(N-n)/(N-1)$ is also undefined (or just infinity) if $N=1$, so neither method has any advantage in that special case; (2) I don't know why you think I am disagreeing that $R_N^2$ is the variance of the empirical distribution when I have stated exactly that in the question; (3) That is question-begging; moreover, the population variance is an unbiased estimator for the superpopulation variance; (4) I have not commented on sampling with replacement here and it is not part of my argument (and I agree that there is an error in the quoted section in this case). $\endgroup$
    – Ben
    Sep 16 at 22:35
  • $\begingroup$ @whuber: Out of interest, do you consider $S_n^2$ to be the "sample variance" and $R_N^2$ the "population variance"? If so, how do you reconcile those disparate treatments? (If not, what alternative treatment do you prefer?) $\endgroup$
    – Ben
    Sep 16 at 22:37
  • $\begingroup$ I confess to being completely lost by your use of "population variance" and "superpopulation variance." I am not trying to discuss terminology -- I rarely introduce "sample variance" and "population variance" unless they are already in use in a context, because I find them potentially confusing -- and I have only done my best to interpret your post; I might have misinterpreted it; but I cannot find mathematical justification for that interpretation. The treatments I prefer are those that are clear and demonstrate their mathematical accuracy. $\endgroup$
    – whuber
    Sep 16 at 23:18
1
$\begingroup$

When we are sampling with replacement then the population size is irrelevant.

Simple counterexample: Imagine an urn model with $k$ red balls, and $k$ blue balls. The probability to draw a red/blue ball is 0.5 independent from the number of total balls $N = 2k$ among the population. So the variance of any sample from that urn should be independent from the size of the population.

It is difficult to find the origin of the error in the text that quoted. It is easy to make little mistakes with $n$ versus $n-1$.

$\endgroup$
3
  • $\begingroup$ That's an interesting perspective. I looked at $N$ more like a way of estimating a way of defining probability of sampling each individual ball, specifically $p(x)=1/N$, so when the $N$ is determined incorrectly the probabilities obtained are not adding up to exactly 1 and therefore estimate of variance is incorrect. $\endgroup$
    – itdxer
    Sep 16 at 13:33
  • $\begingroup$ I also believe there is one small nuance to your example. I believe it's impossible to have the set up with mean 0.5 and odd N. In a way you can say that you're conditioning probabilities on the particular model (i.e. the way balls are being sampled in your example). The problem is that sampling with 0.5 average is impossible when N is odd and since the model is impossible the probabilities which are conditional on the model cannot be defined. Specifically $P(X|M)$ is undefined if $P(M)=0$ (where $M$ is a particular model set up). (Wikipedia: tinyurl.com/y9972et2 ) $\endgroup$
    – itdxer
    Sep 16 at 13:43
  • $\begingroup$ @itdxer whether N is odd or even is irrelevant for my example. I just used the example as a counterexample to show that the formula from your quote is wrong. Whether or not it becomes more complicated in other cases is a different issue, but I used on purpose this simple case where it does not matter. $\endgroup$ Sep 16 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.