Sampling from finite population with replacement

Question

Suppose we have a population of size 10,000. We choose 1000 persons from the population, uniformly and independently, and ask them some question(s). The answers are random variables $X_i$.

Since we choose independently, we possibly can ask the same person multiple times (obtaining the same answers). In other words, we sample with replacement.

My point of view is: $X_i$ are independent and identically distributed random variables, and there is absolutely no difference here with sampling infinite population. For example, WLLN or CLT apply identically in both scenarios (sampling finite population with replacement and sampling infinite population).

Am I correct, or I am missing something?

Answer 1

Suppose 25% of the people in a population of 10,000 favor Proposition A. You are going to poll 1000 of them asking their opinion on the proposition. Suppose you take precautions to sample without replacement (to avoid interviewing anyone twice). Then let the proportion of those interviewed who are in favor be $\hat p_{hyp}.$

By contrast, suppose you take no such precautions (in effect, sampling with replacement). Then let your estimate of the proportion in favor be $\hat p_{bin}.$ Your question is whether there will be an important difference between the estimates $\hat p_{hyp}$ and $\hat p_{bin}.$

The R code below simulates 100,000 such polls according to each method, and summarizes results to simulate $E(\hat p_{hyp})$ and $E(\hat p_{bin})$ and also the standard deviations of these estimates.

set.seed(1111);  m = 10^5; n = 1000
pop = c(rep(1, 2500), rep(0,7500))
p.hyp = replicate(m, sum(sample(pop,n))/n) # sampling without replacement
mean(p.hyp);  sd(p.hyp)
[1] 0.2500288
[1] 0.01297334
p.bin = replicate(m, sum(sample(pop,n,repl=T))/n) # sampling with replacement
mean(p.bin);  sd(p.bin)
[1] 0.2500593
[1] 0.01369168

As you can see, both estimates have expected values very near to the true proportion 25%. The estimate from sampling without replacement is a little less variable.

A general "rule of thumb" is that if you are sampling 10% or less of the population, the difference between sampling with and without replacement can viewed as negligible.

Specifically, if $X \sim \mathsf{Binom}(n, p),$ then $E(X) = np$ and $Var(X) = np(1-p).$ If $Y$ is hypergeometric based on a sample of size $n$ from a population with $S$ successes and $N-S$ failures, then $E(Y) = n(S/N)$ and $Var(Y) = n(S/N)(1-S/N)[(N-n)/(N-1)].$ Letting $p = S/N$ this amounts to $E(Y) = np$ and $Var(Y)=np(1-p)\left(\frac{N-n}{N-1}\right).$

The factor $\frac{N-n}{N-1}$ in $Var(Y)$ is sometimes called the 'finite population correction'; notice that if $n < N/10$ then this factor is close to $1.$ [It's called 'finite population correction' because there is no practical difference between sampling with and without replacement when sampling from an infinite population.]

Of course this factor is not the only distinction between the two distributions. However, for large $n$ and for $p$ not too close to 0 or 1, both the binomial and hypergeometric distributions can be approximated by a normal distribution that matches means and variances. In terms of normal approximations the only distinction between sampling with and without replacement often amounts to the finite population correction.