Does sampling without replacement introduce bias?

If you have a finite population, does sampling without replacement give you biased samples?

• It depends on what you mean by biased: biased with respect to what? Among other things, it depends on what you want to use the sample for.
– Jim
Sep 13, 2016 at 15:32

2 Answers

This depends on what you mean by bias.

Consider a uniform distribution over a population $\{x_1, \dots, x_n\}$, with population mean $\mu := \frac1n \sum_{i=1}^n x_i$.

Let $X_1, \dots, X_m$ be random variables representing each sample without replacement. Then:

• $\DeclareMathOperator{\E}{\mathbb E}\E[X_1] = \mu$, since $X_1$ is just a random sample from the population.
• The conditional expectation $\E[X_2 \mid X_1 = x_j]$ is $$\frac{1}{n-1} \sum_{i \ne j} x_i = \frac{n}{n-1} \mu - \frac{1}{n-1} x_j,$$ which is in general not the same as $\mu$, so $X_2 \mid X_1$ is a biased estimator for $\mu$.
• The unconditional expectation $\E[X_2]$ is \begin{align} \E_{X_1}[ \E[X_2 \mid X_1] ] &= \E_{X_1}\left[ \frac{n}{n-1} \mu - \frac{1}{n-1} X_1 \right] \\&= \frac{n}{n-1} \mu - \frac{1}{n-1} \E X_1 \\&= \frac{n}{n-1} \mu - \frac{1}{n-1} \mu \\&= \mu ,\end{align} so the marginal expectation of $X_2$ is the same as the marginal expectation of $X_1$, and it is thus unbiased in that sense.
• In fact, the marginal distribution of $X_2$ is the same as the distribution of $X_1$: \begin{align} \Pr\left( X_2 = x \right) &= \sum_{x'} \Pr\left( X_1 = x' \right) \Pr\left( X_2 = x \mid X_1 = x' \right) \\&= \sum_{x'} \frac{1}{n} \begin{cases}\frac{1}{n-1} & x \ne x' \\ 0 & x = x'\end{cases} \\&= (n-1) \frac1n \frac{1}{n-1} + (1) \frac1n (0) \\&= \frac1n .\end{align}
• The same argument is also true for $X_3, \dots, X_n$; you can see this either by summing over all the intermediate terms or by induction.
• Just a note for posterity that this is closely related to sampling without replacement being exchangeable. Nov 30, 2020 at 19:45

Sampling the same number of rows as your full dataset without replacement is nothing more than reshuffling the order of the data. Sampling without replacement a small subset is like taking a random draw from the data, again that does not introduce bias.