Do we draw samples with or without replacement when we state the central limit theorem

In central limit theorem (in its most basic flavor), , we say that we draw a large number of samples from an unknown distribution and each time we calculate the mean. Then, for a large sample size, the mean of all these sample observations will tend to be a normal distribution, regardless of the original population distribution.

Now, are these 'samples' drawn with replacement, or without?

• Does it matter? – Neznajka Oct 5 '15 at 21:17

With replacement. In the statement of the theorem, we draw $n$ random sample from a fixed distribution, independently. If we sample without replacement the distribution is constantly changing, which does not meet the requirements of the theorem.
Typically sampling without replacement deals with finite sample spaces, but I suppose it could apply to infinite spaces. My intuition is that in an infinite discrete scenario without replacement, the distribution of $$\sqrt{n}\left(\sum_{i=1}^n \frac{x_i}{n} - \mu \right)$$ would converge in distribution to a normal distribution with mean 0 and variance less than $\sigma^2$.
• Apart from exceptional circumstances, I expect the expression to diverge whenever the support is uncountable, whether or not it is bounded. As an example, let the distribution give probability $1/2$ to $0$ and the rest of its probability to numbers in an interval $(1-\epsilon,1+\epsilon)$ for tiny $\epsilon \lt 1/2$. Suppose $\mu=1/2$. The parenthesized expression converges rapidly to a number in the interval $(1/2-\epsilon,1/2+\epsilon)$. Multiplying it by $\sqrt{n}$ causes that to diverge. – whuber Oct 6 '15 at 0:47