I want to do a permutation test for the significance of a relationship between x and y with a manual function. And I don't want to generate all possible permutations but a sample of them. Before I can compute the p-value I have to generate resamples that are consistent with the null hypothesis of no relationship. My solution for this is the following code:

> resamples <- replicate(n, sample(y,size=length(y),replace=FALSE))

The problem is, that I am not sure, if this is correct. I read in a book on the internet:

"Choose permutation resamples from the data without replacement".

Is it false if the observations (of one sample) are chosen without replacement but not my resamples. Because my function above generates sometimes the same resamples by chance.

  • $\begingroup$ It is unclear what is your question, could you clarify? What is false? What is your question about? Function sample takes a random sample from a vector of values so it can return duplicated values. $\endgroup$
    – Tim
    May 15, 2016 at 6:58
  • 1
    $\begingroup$ The phrase "without replacement": is it related to the observations of one resample or means "without replacement" that all resamples have to be different. $\endgroup$
    – Chris
    May 15, 2016 at 8:11
  • $\begingroup$ I think you should clarify whether this question is primarily about coding (in which case it would be off-topic here but we could migrate it to Stack Overflow for you - have a look at our help center for more details about what questions are within the scope of our site) or whether your main issue is the underlying statistical one. For what it's worth, I think there is enough statistical content in this question to justify it staying on CV. $\endgroup$
    – Silverfish
    May 15, 2016 at 10:23
  • 2
    $\begingroup$ There is a distinction to be made between bootstrapping and a permutation test: Bootstrapping samples with replacement. A permutation test just shuffles the data in y relative to x so as to break the pairing / relationship between them; for a permutation test, you sample with replacement. Your code is appropriate for a permutation test, but it does not implement bootstrapping. $\endgroup$ May 15, 2016 at 11:48

1 Answer 1


You ask about meaning of sampling with replacement. Imagine that you have some population that you sample from. When you sample without replacement, then each time you randomly sample a case from this population it is removed from the population, so it cannot be sampled any more. When sampling with replacement, each time a case is randomly sampled from the population afterwards it is returned back to the population, so it possibly can be sampled again.

This is nicely described in Wikipedia article about simple random sampling:

In small populations and often in large ones, such sampling is typically done "without replacement", i.e., one deliberately avoids choosing any member of the population more than once. Although simple random sampling can be conducted with replacement instead, this is less common and would normally be described more fully as simple random sampling with replacement. Sampling done without replacement is no longer independent, but still satisfies exchangeability, hence many results still hold. Further, for a small sample from a large population, sampling without replacement is approximately the same as sampling with replacement, since the odds of choosing the same individual twice is low.

As you correctly noticed, in R's sample function there is parameter replace to declare if you want to sample with (TRUE), or without (FALSE) replacement.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.