I recently learned about doing hypothesis tests by bootstrapping and permutation. I'm trying to relate it to other traditional hypothesis testing such as t-test or chi-square test. Are they in the same category? Can I think of bootstrap hypothesis testing as a non-parametric approach?

  • $\begingroup$ Can you provide a citation or example? Bootstrapping is generally a method for obtaining confidence intervals and, to my knowledge, is not a specific formulation or approach to hypothesis testing. $\endgroup$
    – Todd D
    Commented Jul 21, 2018 at 2:22
  • $\begingroup$ Similar interesting Q&A, and last comment: stats.stackexchange.com/questions/41683/… $\endgroup$
    – Rob
    Commented Jul 21, 2018 at 3:22
  • $\begingroup$ @Todd You're correct. But we can also use it to test a hypothesis. For example, we have a dataset of height measurements for kids and adults. We compute the mean of the data and then split the dataset into kids' heights and adults' heights. Calculate the means of each split and the their difference. Then shift the means of both splits to the mean of the original dataset. Perform bootstrapping to construct the confidence interval of mean difference. Then check the probability of the mean difference of the two splits from CI. $\endgroup$
    – ChasNew
    Commented Jul 21, 2018 at 4:41

2 Answers 2


Bootstrap-based tests and permutation tests appear to be similar procedures (resampling with replacement and resampling without replacement), but have different foundations and different strengths and weaknesses.

As the Wikipedia page says, "bootstrapping is the practice of estimating properties of an estimator (such as its variance) by measuring those properties when sampling from an approximating distribution", where the simplest case of the approximating distribution is given by the sample itself. Say you have a sample statistic which you intend as an estimator of a distribution property. Bootstrapping allows you to estimate e.g. the bias and variance of this estimator. This can be extended to find approximate confidence intervals for the underlying distribution property. Since null hypothesis tests and confidence intervals are complementary, this can be used to construct a bootstrap-based null hypothesis test. The bootstrap is very versatile because it does not need (strong) distributional assumptions, but because it relies on an approximating distribution it is an approximate procedure.

Permutation tests rely on symmetries of a distribution under the null hypothesis, specifically exchangeability, meaning that the distribution of the data under the null hypothesis does not change when sample values are exchanged (e.g. between two different samples). If the null hypothesis you want to test has this property of exchangeability, the resulting permutation test is exact. This is the same property as that of standard parametric tests, with the difference that parametric tests need stronger assumptions than that of exchangeability. In this sense, permutation tests and parametric tests are "in the same category", namely that of exact tests.

Both bootstrap-based tests and permutation test make weaker assumptions than parametric tests, and in this sense they may both be called nonparametric.


Bootstrapping is a way to construct the distribution of model parameters or functions of model parameters under a certain hypothesis; this hypothesis is often independence or lack of effect. For standard NHST, the null hypothesis distributions (or reasonable approximations) have been derived theoretically. Bootstrapping allows you to approximate a null hypothesis distribution for values that don't have theoretical/standard sampling distributions. By permuting or sampling the independent variables and the response, you are simulating new data sets in which there is no underlying relationship since you are assigning correspondence at random. Calculating the values of interest from these datasets gives you samples from the null distribution and these samples collectively are used to approximate percentiles for both CIs and hypothesis testing.

  • 1
    $\begingroup$ Maybe it would be helpful to explain how the null distribution is approximated by the bootstrap. $\endgroup$
    – Michael M
    Commented Jul 21, 2018 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.