22
$\begingroup$

Sampling with replacement has two advantages over sampling without replacement as I see it:

1) You don't need to worry about the finite population correction.

2) There is a chance that elements from the population are drawn multiple times - then you can recycle the measurements and save time.

Of course from an academic POV one has to investigate both methods. But from a practical POV I don't see why one would consider sampling without replacement given the advantages of with replacement.

But I am a beginner in statistics so there might be plenty of good reasons why without replacement might be the superior choice - at least for specific use cases. Please, unconfuse me!

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Hint: Consider what the effect of applying the finite population correction is, & why that might be advantageous. (Also note that (1) doing sums is almost always less trouble & expense than collecting data; (2) if you can distinguish individuals you shouldn't "recycle" measurements, but base inference only on the distinct individuals.) $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Commented Sep 11, 2013 at 15:15
  • $\begingroup$ Honestly, I do not actually understand any of your assertions. The FPC compensates the numerical consequences of lack of independence of the measurements. But I don't know why this is advantageous. (1) how does this relate to my question? (2) Why "shouldn't" you recycle a measurement? Isn't doing so the direct logical consequence of having coincidentally drawn twice the same item when sampling with replacement? $\endgroup$
    – Raffael
    Commented Sep 11, 2013 at 15:24

5 Answers 5

Reset to default
20
$\begingroup$

Expanding on the answer of @Scortchi . . .

Suppose the population had 5 members and you have budget to sample 5 individuals. You are interested in the population mean of a variable X, a characteristic of individuals in this population. You could do it your way, and randomly sample with replacement. The variance of the sample mean will be V(X)/5.

On the other hand, suppose you sample the five individuals without replacement. Then, the variance of the sample mean is 0. You've sampled the whole population, each individual exactly once, so there is no distinction between "sample mean" and "population mean." They are the same thing.

In the real world, you should jump for joy each time you have to do the finite population correction because (drumroll . . .) it makes the variance of your estimator go down without you having to collect more data. Almost nothing does this. It's like magic: good magic.

Saying the exact same thing in math (pay attention to the <, and assume sample size is greater than 1): \begin{equation} \textrm{finite sample correction} = \frac{N-n}{N-1} < \frac{N-1}{N-1} = 1 \end{equation}

Correction < 1 means that applying the correction makes the variance go DOWN, 'cause you apply the correction by multiplying it against the variance. Variance DOWN == good.

Moving in the opposite direction, entirely away from math, think about what you are asking. If you want to learn about the population and you can sample 5 people from it, does it seem likely that you will learn more by taking the chance of sampling the same guy 5 times or does it seem more likely that you will learn more by ensuring that you sample 5 different guys?

The real world case is almost the opposite of what you are saying. Almost never do you sample with replacement --- it's only when you are doing special things like bootstrapping. In that case, you are actually trying to screw up the estimator and give it a "too big" variance.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Under "bootstrapping" I understand using a parameter of the sample in place of the parameter of the population (which you actually would have had to use) to estimate a parameter of the population. Why would you be interested to "screw up" the estimator and give it a "too big" variance? $\endgroup$
    – Raffael
    Commented Sep 11, 2013 at 17:54
  • 1
    $\begingroup$ @Яaffael I am talking about non-parametric bootstrapping. You take your sample (say of size 100), re-sample from it with replacement (100 times yielding a bootstrap sample of size 100), and then re-calculate your estimator of interest. You are treating the sample as a toy population, simulating drawing a sample from it, calculating an estimator. If you sampled from the toy population without replacement, you would exactly copy the toy population in the sample, getting the original estimate as the new estimate (ie variance=0). To avoid this, so you sample with replacement. $\endgroup$
    – Bill
    Commented Sep 12, 2013 at 12:34
6
$\begingroup$

The precision of estimates is usually higher for sampling without replacement comparing to sampling with replacement.

For example, it is possible to select only one element $n$ times when sampling is done with replacement in an extreme case. That could lead to very imprecise estimate of the population parameter of interest. Such a situation is not possible under sampling without replacement. So the variance is usually lower for estimates made from sampling without replacement.

Improve this answer
$\endgroup$
5
$\begingroup$

I don't think the answers here are totally adequate, and they seem to argue for the limiting case in which your amount of data is very low.

With a sufficiently large sample, this isn't a worry at all, especially with many bootstrap resamples (~1000). If I have sampled from the true distribution a dataset of size 10,000, and I resample with replacement 1,000 times, then the variance I gain (as opposed to the variance I would obtain by doing no replacement) is totally negligible.

I would say that the more accurate answer is this: resampling without replacement is essential when estimating the confidence of a second-order statistic. For example, if I'm using a bootstrap to estimate the uncertainty that I have in a dispersion measurement. Drawing with replacement for such a quantity can artificially bias the recovered dispersions low.

For a concrete example with real data, if you're up to it, see this paper https://arxiv.org/abs/1612.02827

it briefly discusses your question on page 10

Improve this answer
$\endgroup$
1
$\begingroup$

But from a practical POV I don't see why one would consider sampling without replacement given the advantages of with replacement.

In practice, sampling without replacement saves you the need to, well, make replacements. This has two benefits:

  1. You can just take a larger sample and consider it as multiple individual samples.

  2. Replacements in the real world can be costly, time-consuming or even nigh-impossible.

    Suppose you're a powerful alien sampling people from the Earth's human population by abducting them. Do you realize how hard it is to perform a replacement? You would have to erase their memory, cook up excuses for their absence, etc. Much better to just abduct a bunch of them and base your statistical analysis on that.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Just wanted to say that I really liked the analogy at the end :) $\endgroup$
    – FafaDog
    Commented Jan 25, 2023 at 15:53
0
$\begingroup$

I have a result which treats without replacement practically as with replacement and removes all the difficulties. Note that with replacement calculations are much easier. So, if a probability involves p and q,probabilities of success and failure, in with replacement case, the corresponding probability in without replacement case is obtained simply with the the replacement of p^a.q^b with (N-a-b)C(R-a) for any a and b, where N, R are the total number of balls and the number of white balls. Remember that p is treated as R/N.

K.Balasubramanian

Improve this answer
$\endgroup$
1
  • $\begingroup$ there was an omission. (N-a-b)C(R-a)/(NCR) is the correct expression. For example the mean np becomes n(N-1-0)/(R-1) /NCR. you can check any such result. $\endgroup$
    – Krish Balasubramanian
    Commented Jul 17, 2017 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.