# Bootstrap vs. permutation hypothesis testing

There are several popular resampling techniques, which are often used in practice, such as bootstrapping, permutation test, jackknife, etc. There are numerous articles & books discuss these techniques, for example Philip I Good (2010) Permutation, Parametric, and Bootstrap Tests of Hypotheses

My question is which resampling technique has gained the more popularity and easier to implement? Bootstrapping or permutation tests?

• Popularity is hardly a good measure of quality. Judging by the number of citations (customers), McDonalds is a far more popular (better?) restaurant than any three-star Michelin establishment. Will you take your next seminar speaker to McDonalds, then? Mar 3 '15 at 17:27

Both are popular and useful, but primarily for different uses. The permutation test is best for testing hypotheses and bootstrapping is best for estimating confidence intervals.

Permutation tests test a specific null hypothesis of exchangeability, i.e. that only the random sampling/randomization explains the difference seen. This is the common case for things like t-tests and ANOVA. It can also be expanded to things like time series (null hypothesis that there is no serial correlation) or regression (null hypothesis of no relationship). Permutation tests can be used to create confidence intervals, but it requires many more assumptions, that may or may not be reasonable (so other methods are preferred). The Mann-Whitney/Wilcoxon test is actually a special case of a permutation test, so they are much more popular than some realize.

The bootstrap estimates the variability of the sampling process and works well for estimating confidence intervals. You can do a test of hypothesis this way but it tends to be less powerful than the permutation test for cases that the permutation test assumptions hold.

• Thanks for the answer. Why is the bootstrap confidence interval less powerful than the permutation test? How much so? Can one characterize the situations under which it is significantly less powerful? It seems an advantage to be able to show a confidence interval, so in that sense the bootstrap seems more valuable. Dec 27 '14 at 18:22
• @dfrankow, the 2 methods use different assumptions. For large samples and differences they will both be fine, but with smaller samples/differences the permutation test is more likely to find differences and be appropriate. See this answer: stats.stackexchange.com/questions/112147/… for examples where the bootstrap is not even correctly sized (rejects too often when the null is true). Dec 29 '14 at 17:58
• Isn't a permutation test a variation on bootstrapping? Sep 28 '19 at 22:50
• @VickiB, Bootstrapping and Permutation tests are often mentioned together, but boostrapping samples with replacement and permutation samples without replacement which makes a difference in what they can do and how powerful they are. Sep 30 '19 at 17:00
• I think more important is that the bootstrap samples (approximately) from the actual data generating distribution but permutation tests sample from a different data distribution that satisifies the null hypothesis. Jun 10 '20 at 3:49

If you are using R, then they are all easy to implement. See, for instance, http://www.burns-stat.com/pages/Tutor/bootstrap_resampling.html

I would say there is a third major technique: cross validation. This is used to test the predictive power of models.

My question is which resampling technique has gained the more popularity
Bootstrapping or permutation tests?

1. Bootstrapping is mostly about generating large sample standard errors or confidence intervals; permutation tests as the name suggests are mostly about testing. (Each can be adapted to be used for the other task though.)

2. How would we judge popularity? If we look at fields like psychology and education we can find plenty of use of rank based tests like Wilcoxon-Mann-Whitney, the signed rank test, rank-correlation tests and so on. These are all permutation tests (on the other hand there are many instances where permutation tests of the original data could be used instead but are usually not). In some other application areas, permutation tests would rarely be used, but the varying popularity across application areas sometimes says more about the local culture of whichever area than usefulness.

easier to implement?

In many cases - especially simpler ones - they're almost exactly equally easy -- it's essentially the difference between sampling with replacement and sampling without replacement.

In some of the more complex cases, bootstrapping is easier to do because (looking at it from the testing point of view) it operates as well under the alternative as the null (at least naive implementations will be -- doing it so that it works well may be much more complicated).

Exact permutation tests can be difficult in the more complex cases because a suitable exchangable quantity may be unobservable -- often a nearly-exchangeable quantity may be substituted at the price of exactness (and of being truly distribution-free).

Bootstrapping essentially gives up on the corresponding exactness criterion (e.g. exact coverage of intervals) from the outset, and instead focuses on trying to get reasonably good coverage in large samples (sometimes with less success than may be understood; if you haven't checked, don't assume your bootstrap gives the coverage you expect it to).

Permutation tests can work on small samples (though limited choice of significance levels can sometimes be a problem with very small samples), while the bootstrap is a large-sample technique (if you use it with small samples, in many cases the results may not be very useful).

I rarely see them as competitors on the same problem, and have used them on (different) real problems -- often there will be a natural choice of which to look at.

There are benefits to both, but neither in a panacaea. If you're hoping to reduce learning effort by focusing on only one of them you're likely to be disappointed -- both are essential parts of the resampling toolbox.

• Could you please clarify what "a suitable exchangable quantity may be unobservable" means? (+1 obviously) Mar 22 '19 at 22:02
• Consider trying to conduct a permutation test in an experiment with two factors and a covariate (or just consider a regression with several predictors). With independence and under a null of no effects at all, the observations are exchangeable and you can therefore test that hypothesis but you don't have a way to construct a permutation test of just the factors (since you expect the covariate to have an effect and testing it being null isn't interesting); similarly you can't construct a permutation test of just one of the two factors. ... ctd Mar 23 '19 at 23:22
• ctd... There's an obvious exchangeable quantity if you know the population coefficients you aren't testing (and the errors would always be exchangeable) but you can't observe those things. If you substitute estimates of the coefficients or of the errors (i.e. the residuals) the quantities are longer exchangeable. However under some particular conditions they would be approximately exchangeable (some people advocate doing precisely this) .... and if you do that you end up with something akin to a bootstrap but with sampling without replacement instead of sampling with replacement. Mar 23 '19 at 23:26
• Thank you; I will think over this carefully. I suspect there something deeper for me to learn here. :) Mar 24 '19 at 0:58
• @NULL for some reason I missed your request for a reference. For a starting point, some of the references here should do: davegiles.blogspot.com/2019/04/… Sep 29 '19 at 9:18