# Do we draw samples with or without replacement when we state the central limit theorem

In central limit theorem (in its most basic flavor), , we say that we draw a large number of samples from an unknown distribution and each time we calculate the mean. Then, for a large sample size, the mean of all these sample observations will tend to be a normal distribution, regardless of the original population distribution.

Now, are these 'samples' drawn with replacement, or without?

• Does it matter? Oct 5, 2015 at 21:17

With replacement. In the statement of the theorem, we draw $n$ random sample from a fixed distribution, independently. If we sample without replacement the distribution is constantly changing, which does not meet the requirements of the theorem.

Typically sampling without replacement deals with finite sample spaces, but I suppose it could apply to infinite spaces. My intuition is that in an infinite discrete scenario without replacement, the distribution of $$\sqrt{n}\left(\sum_{i=1}^n \frac{x_i}{n} - \mu \right)$$ would converge in distribution to a normal distribution with mean 0 and variance less than $\sigma^2$.

• If the support of the distribution is unbounded--as for a geometric or Poisson distribution, for instance--then I think that expression should not converge at all.
– whuber
Oct 6, 2015 at 0:31
• @whuber great point! And quite easy to prove! Oct 6, 2015 at 0:37
• If the support is bounded, then do you believe it converges with tighter variance? Oct 6, 2015 at 0:38
• Apart from exceptional circumstances, I expect the expression to diverge whenever the support is uncountable, whether or not it is bounded. As an example, let the distribution give probability $1/2$ to $0$ and the rest of its probability to numbers in an interval $(1-\epsilon,1+\epsilon)$ for tiny $\epsilon \lt 1/2$. Suppose $\mu=1/2$. The parenthesized expression converges rapidly to a number in the interval $(1/2-\epsilon,1/2+\epsilon)$. Multiplying it by $\sqrt{n}$ causes that to diverge.
– whuber
Oct 6, 2015 at 0:47
• @whuber good point. Sampling without replacement should be done in moderation. Almost never infinitely! Oct 6, 2015 at 0:52