What is the difference between bootstrap hypothesis testing/permutation test and traditional hypothesis testing?

Question

I recently learned about doing hypothesis tests by bootstrapping and permutation. I'm trying to relate it to other traditional hypothesis testing such as t-test or chi-square test. Are they in the same category? Can I think of bootstrap hypothesis testing as a non-parametric approach?

Answer 1

Bootstrap-based tests and permutation tests appear to be similar procedures (resampling with replacement and resampling without replacement), but have different foundations and different strengths and weaknesses.

As the Wikipedia page says, "bootstrapping is the practice of estimating properties of an estimator (such as its variance) by measuring those properties when sampling from an approximating distribution", where the simplest case of the approximating distribution is given by the sample itself. Say you have a sample statistic which you intend as an estimator of a distribution property. Bootstrapping allows you to estimate e.g. the bias and variance of this estimator. This can be extended to find approximate confidence intervals for the underlying distribution property. Since null hypothesis tests and confidence intervals are complementary, this can be used to construct a bootstrap-based null hypothesis test. The bootstrap is very versatile because it does not need (strong) distributional assumptions, but because it relies on an approximating distribution it is an approximate procedure.

Permutation tests rely on symmetries of a distribution under the null hypothesis, specifically exchangeability, meaning that the distribution of the data under the null hypothesis does not change when sample values are exchanged (e.g. between two different samples). If the null hypothesis you want to test has this property of exchangeability, the resulting permutation test is exact. This is the same property as that of standard parametric tests, with the difference that parametric tests need stronger assumptions than that of exchangeability. In this sense, permutation tests and parametric tests are "in the same category", namely that of exact tests.

Both bootstrap-based tests and permutation test make weaker assumptions than parametric tests, and in this sense they may both be called nonparametric.

Answer 2

Bootstrapping is a way to construct the distribution of model parameters or functions of model parameters under a certain hypothesis; this hypothesis is often independence or lack of effect. For standard NHST, the null hypothesis distributions (or reasonable approximations) have been derived theoretically. Bootstrapping allows you to approximate a null hypothesis distribution for values that don't have theoretical/standard sampling distributions. By permuting or sampling the independent variables and the response, you are simulating new data sets in which there is no underlying relationship since you are assigning correspondence at random. Calculating the values of interest from these datasets gives you samples from the null distribution and these samples collectively are used to approximate percentiles for both CIs and hypothesis testing.