3
$\begingroup$

If you have a finite population, does sampling without replacement give you biased samples?

$\endgroup$
1
  • $\begingroup$ It depends on what you mean by biased: biased with respect to what? Among other things, it depends on what you want to use the sample for. $\endgroup$
    – Jim
    Sep 13, 2016 at 15:32

2 Answers 2

4
$\begingroup$

This depends on what you mean by bias.

Consider a uniform distribution over a population $\{x_1, \dots, x_n\}$, with population mean $\mu := \frac1n \sum_{i=1}^n x_i$.

Let $X_1, \dots, X_m$ be random variables representing each sample without replacement. Then:

  • $\DeclareMathOperator{\E}{\mathbb E}\E[X_1] = \mu$, since $X_1$ is just a random sample from the population.
  • The conditional expectation $\E[X_2 \mid X_1 = x_j]$ is $$\frac{1}{n-1} \sum_{i \ne j} x_i = \frac{n}{n-1} \mu - \frac{1}{n-1} x_j,$$ which is in general not the same as $\mu$, so $X_2 \mid X_1$ is a biased estimator for $\mu$.
  • The unconditional expectation $\E[X_2]$ is \begin{align} \E_{X_1}[ \E[X_2 \mid X_1] ] &= \E_{X_1}\left[ \frac{n}{n-1} \mu - \frac{1}{n-1} X_1 \right] \\&= \frac{n}{n-1} \mu - \frac{1}{n-1} \E X_1 \\&= \frac{n}{n-1} \mu - \frac{1}{n-1} \mu \\&= \mu ,\end{align} so the marginal expectation of $X_2$ is the same as the marginal expectation of $X_1$, and it is thus unbiased in that sense.
  • In fact, the marginal distribution of $X_2$ is the same as the distribution of $X_1$: \begin{align} \Pr\left( X_2 = x \right) &= \sum_{x'} \Pr\left( X_1 = x' \right) \Pr\left( X_2 = x \mid X_1 = x' \right) \\&= \sum_{x'} \frac{1}{n} \begin{cases}\frac{1}{n-1} & x \ne x' \\ 0 & x = x'\end{cases} \\&= (n-1) \frac1n \frac{1}{n-1} + (1) \frac1n (0) \\&= \frac1n .\end{align}
  • The same argument is also true for $X_3, \dots, X_n$; you can see this either by summing over all the intermediate terms or by induction.
$\endgroup$
1
  • $\begingroup$ Just a note for posterity that this is closely related to sampling without replacement being exchangeable. $\endgroup$
    – Danica
    Nov 30, 2020 at 19:45
3
$\begingroup$

Sampling the same number of rows as your full dataset without replacement is nothing more than reshuffling the order of the data. Sampling without replacement a small subset is like taking a random draw from the data, again that does not introduce bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.