1
$\begingroup$

In various scattered places online and StackExchange I've read that bootstrap resampling is more appropriate for calculating confidence intervals while permutation resampling is more appropriate for hypothesis testing, but why is this so (or is this generalization incorrect)?

I've seen it argued that this is because bootstrap resampling estimates the true population rather than the distribution under the null, but it seems to me that if confidence interval calculation is valid, the hypothesis testing is also appropriate via seeing if the null hypothesis value lies within the interval. Moreover, we can emulate the null distribution in bootstrapping by shifting our means (subtracting out the sample means and then adding back in the mean under the null).

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Simply: The non-parametric bootstrap resampling generates datasets under the alternative hypothesis. Permutation resampling generates datasets under the null hypothesis. The CI probabilities are conditional upon the original sample estimate being the truth. The p-value probability is conditional upon the null hypothesis being the truth. Both procedures can perform the other's probability calculations with some simple to very complex modifications.

Improve this answer
$\endgroup$
4
  • $\begingroup$ How does this address the pre-emption I raised that you can simulate the null distribution in bootstrapping fairly simply by shifting the means? $\endgroup$
    – Josh
    Commented Sep 26, 2019 at 15:04
  • $\begingroup$ @Josh Because you can't do that for non-pivotal quantities. $\endgroup$
    – AdamO
    Commented Sep 26, 2019 at 15:07
  • $\begingroup$ I'm unfortunately ignorant as to what that ([not] for non-pivotal quantities) means :( $\endgroup$
    – Josh
    Commented Sep 26, 2019 at 15:11
  • $\begingroup$ Actually I checked out the link @AdamO posted in the other comments on different bootstrap types which mentioned non-pivotal-ableness. (stats.stackexchange.com/questions/20701/…) Which is quite helpful in understanding this issue also $\endgroup$
    – Josh
    Commented Sep 27, 2019 at 19:57
1
$\begingroup$

Permutation tests address nonparametric hypotheses such as equal distributions of two samples or independence of two variables. These tests are exact in theory, and in practice approximation by Monte Carlo simulation is normally unbiased and very precise. However, by nature they do not address parametric problems. It is true that there are parametric versions of equality and independence problems and permutation tests can be adapted to them. This includes computing confidence intervals, but these are cumbersome and do normally involve more assumptions than the permutation test would otherwise need.

Bootstrap tests on the other hand are usually biased. If a bootstrap and a proper permutation test can be set up for the same problem, there are in most cases good reasons to prefer the permutation test. However there are test problems for which there are no permutation tests (without tedious adaptation that may lose some of the original advantages) but that can be treated using bootstrap.

On the other hand, as permutation tests are by nature nonparametric, they don't lend themselves easily to confidence intervals.

So in general it's wrong to say that bootstrap is always preferable for CIs and permutation tests are always better for tests, however where a permutation test can be done it is usually better than a bootstrap test, and permutation CIs are tedious and problematic if available at all.

Improve this answer
$\endgroup$
6
  • $\begingroup$ that made a ton of sense, particularly that last paragraph. I read it as Boostrap CIs are sometimes "preferred" more for their flexibility and ease of implementation given the other available options rather than their precision. I would be interested in more on what Bootstraps are more biased that permutation resampling if possible. $\endgroup$
    – Josh
    Commented Sep 26, 2019 at 15:10
  • $\begingroup$ I can't go into detail here because this depends strongly on the precise parameter, estimator, testing problem. Bootstrap implicitly assumes that the observed empirical distribution is a perfect representation of the underlying distribution (at least regarding crucial aspects for the problem at hand), which is often true for sample sizes tending to infinity but usually not for finite samples. How this plays out depends critically on what exactly is done; often it is quite harmless, sometimes not. $\endgroup$
    – Christian Hennig
    Commented Sep 26, 2019 at 16:04
  • $\begingroup$ This answer is wrong on many counts. Permutation tests have nothing to do with strong vs weak hypotheses. "Permuting" merely gives you a bunch of datasets simulating a null. Test for a two-sample mean difference using the sample mean, and you can "address" the parametric problem of estimating the center of a normal distribution. If you calculate the KS test statistic in those datasets, you can claim to test for a difference is CDFs, but the same hypothesis comes about using the non-resampled version of the test. The wrong bootstrap test will be biased, but correct versions are available. $\endgroup$
    – AdamO
    Commented Sep 26, 2019 at 18:48
  • $\begingroup$ The designation of the permutation test as "nonparmetric" has nothing to do with it's ability to "lend itself to confidence intervals". If anything, that's more to do with the fact it's a test and not an estimator. Still, the non-parametric Fisher's Exact test can easily be inverted to find confidence intervals. The non-parametric bootstrap is conceivably the gold standard example of a robust way to calculate confidence intervals. Examples of bootstrap tests here: stats.stackexchange.com/questions/20701/… $\endgroup$
    – AdamO
    Commented Sep 26, 2019 at 18:52
  • $\begingroup$ I haven't said that these things can't be done; particularly I know that permutation based CIs can be constructed and they may well be fine for some problems. As far as I have seen theory for bootstrap in finite samples, there is always a bias term (normally going to zero with n to infinity). True, this can be reduced by more sophisticated/correct versions, but it can not normally be reduced to zero. Which of course doesn't mean that bootstrap isn't "available"; we can live with finite sample bias for many asymptotically motivated methods. $\endgroup$
    – Christian Hennig
    Commented Sep 26, 2019 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.