2
$\begingroup$

Suppose we have a population of size 10,000. We choose 1000 persons from the population, uniformly and independently, and ask them some question(s). The answers are random variables $X_i$.

Since we choose independently, we possibly can ask the same person multiple times (obtaining the same answers). In other words, we sample with replacement.

My point of view is: $X_i$ are independent and identically distributed random variables, and there is absolutely no difference here with sampling infinite population. For example, WLLN or CLT apply identically in both scenarios (sampling finite population with replacement and sampling infinite population).

Am I correct, or I am missing something?

$\endgroup$
  • $\begingroup$ (1) In your example sampling $n = 1000$ from among $N = 10,000,$ there is not much difference between sampling with and without replacement. You are sampling only 10% of the population and interviewing a given individual twice will be relatively rare. (2) However, in sampling without replacement, the $X_i$ are not independent. For example, in sampling 13 cards from a deck of 52, the probability that the first card dealt is an Ace is $4/52,$ but then the probability of getting an Ace on the 2nd card is $3/51.$ ... $\endgroup$ – BruceET Nov 11 '18 at 8:50
  • $\begingroup$ ... (3) For sampling without replacement, use the hypergeometric distribution; for sampling with replacement, use the binomial distribution. (4) In sampling 13 cards from a deck of 52 without replacement, the probability of getting four Aces is ${4 \choose 4}{48 \choose 9}/{52 \choose 13} =.00264.$ If you were sampling with replacement, the probability of getting four Aces would be ${13 \choose 4}(1/13)^4(12/13)^9 = .01218.$ // In R, dhyper(4, 4, 48, 13) returns 0.002641056; dbinom(4, 13, 1/13) returns 0.01218074. $\endgroup$ – BruceET Nov 11 '18 at 9:07
  • $\begingroup$ @BruceET Thank you; I am not interested in sampling without replacement. I am interested in whether finite population size affects in any way sampling with replacement, because I've heard people saying it does. Say I sample 1000 times, the population size can be 10, or 1000, or 10000, or infinite. I apply CLT to find the distribution of sum of samples, and I think the accuracy of using CLT will be exactly the same, no matter what is the population size. $\endgroup$ – kludg Nov 11 '18 at 9:31
  • $\begingroup$ Well, to discuss the distinction between the two kinds of sampling you need to be interested in sampling without replacement --- just temporarily. What you say about what 'people say' is a little garbled. See my Answer on both accounts. $\endgroup$ – BruceET Nov 11 '18 at 10:42
  • $\begingroup$ Do you record the instances where $X_i$ and $X_j$ refer to the same individual, or do they go unnoticed? $\endgroup$ – kjetil b halvorsen Nov 11 '18 at 15:38
4
$\begingroup$

Suppose 25% of the people in a population of 10,000 favor Proposition A. You are going to poll 1000 of them asking their opinion on the proposition. Suppose you take precautions to sample without replacement (to avoid interviewing anyone twice). Then let the proportion of those interviewed who are in favor be $\hat p_{hyp}.$

By contrast, suppose you take no such precautions (in effect, sampling with replacement). Then let your estimate of the proportion in favor be $\hat p_{bin}.$ Your question is whether there will be an important difference between the estimates $\hat p_{hyp}$ and $\hat p_{bin}.$

The R code below simulates 100,000 such polls according to each method, and summarizes results to simulate $E(\hat p_{hyp})$ and $E(\hat p_{bin})$ and also the standard deviations of these estimates.

set.seed(1111);  m = 10^5; n = 1000
pop = c(rep(1, 2500), rep(0,7500))
p.hyp = replicate(m, sum(sample(pop,n))/n) # sampling without replacement
mean(p.hyp);  sd(p.hyp)
[1] 0.2500288
[1] 0.01297334
p.bin = replicate(m, sum(sample(pop,n,repl=T))/n) # sampling with replacement
mean(p.bin);  sd(p.bin)
[1] 0.2500593
[1] 0.01369168

As you can see, both estimates have expected values very near to the true proportion 25%. The estimate from sampling without replacement is a little less variable.

A general "rule of thumb" is that if you are sampling 10% or less of the population, the difference between sampling with and without replacement can viewed as negligible.

Specifically, if $X \sim \mathsf{Binom}(n, p),$ then $E(X) = np$ and $Var(X) = np(1-p).$ If $Y$ is hypergeometric based on a sample of size $n$ from a population with $S$ successes and $N-S$ failures, then $E(Y) = n(S/N)$ and $Var(Y) = n(S/N)(1-S/N)[(N-n)/(N-1)].$ Letting $p = S/N$ this amounts to $E(Y) = np$ and $Var(Y)=np(1-p)\left(\frac{N-n}{N-1}\right).$

The factor $\frac{N-n}{N-1}$ in $Var(Y)$ is sometimes called the 'finite population correction'; notice that if $n < N/10$ then this factor is close to $1.$ [It's called 'finite population correction' because there is no practical difference between sampling with and without replacement when sampling from an infinite population.]

Of course this factor is not the only distinction between the two distributions. However, for large $n$ and for $p$ not too close to 0 or 1, both the binomial and hypergeometric distributions can be approximated by a normal distribution that matches means and variances. In terms of normal approximations the only distinction between sampling with and without replacement often amounts to the finite population correction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.