Does sampling without replacement introduce bias? If you have a finite population, does sampling without replacement give you biased samples?
 A: This depends on what you mean by bias.
Consider a uniform distribution over a population $\{x_1, \dots, x_n\}$, with population mean $\mu := \frac1n \sum_{i=1}^n x_i$.
Let $X_1, \dots, X_m$ be random variables representing each sample without replacement.
Then:


*

*$\DeclareMathOperator{\E}{\mathbb E}\E[X_1] = \mu$, since $X_1$ is just a random sample from the population.

*The conditional expectation $\E[X_2 \mid X_1 = x_j]$ is
$$\frac{1}{n-1} \sum_{i \ne j} x_i = \frac{n}{n-1} \mu - \frac{1}{n-1} x_j,$$
which is in general not the same as $\mu$, so $X_2 \mid X_1$ is a biased estimator for $\mu$.

*The unconditional expectation $\E[X_2]$ is
\begin{align}
\E_{X_1}[ \E[X_2 \mid X_1] ]
&= \E_{X_1}\left[ \frac{n}{n-1} \mu - \frac{1}{n-1} X_1 \right]
\\&= \frac{n}{n-1} \mu - \frac{1}{n-1} \E X_1
\\&= \frac{n}{n-1} \mu - \frac{1}{n-1} \mu
\\&= \mu
,\end{align}
so the marginal expectation of $X_2$ is the same as the marginal expectation of $X_1$, and it is thus unbiased in that sense.

*In fact, the marginal distribution of $X_2$ is the same as the distribution of $X_1$:
\begin{align}
\Pr\left( X_2 = x \right)
&= \sum_{x'} \Pr\left( X_1 = x' \right) \Pr\left( X_2 = x \mid X_1 = x' \right)
\\&= \sum_{x'} \frac{1}{n} \begin{cases}\frac{1}{n-1} & x \ne x' \\ 0 & x = x'\end{cases}
\\&= (n-1) \frac1n \frac{1}{n-1} + (1) \frac1n (0)
\\&= \frac1n
.\end{align}

*The same argument is also true for $X_3, \dots, X_n$; you can see this either by summing over all the intermediate terms or by induction.

A: Sampling the same number of rows as your full dataset without replacement is nothing more than reshuffling the order of the data. Sampling without replacement a small subset is like taking a random draw from the data, again that does not introduce bias.
