Can we calculate a confidence interval using a permutation testing approach?

Question

Tim Hesterberg's "What Teachers Should Know about the Bootstrap" is excellent. On page 77 he writes: "We cannot use permutation testing to obtain confidence intervals."

Is that correct? If so, can someone explain and give a citation.

Introduction to Statistical Investigations by Tintle et al (also excellent) suggests that we can use permutation tests in this way:

SECTION 3.1 Summary

Once a sample proportion has been determined to differ significantly from a hypothesized value for the long-run process probability [using a permutation test], a typical next step is to estimate the value of the long-run process probability with an interval of values, the confidence interval for the parameter. Or our research question may be phrased such that a confidence interval is the more appropriate tool (e.g., “what is the probability that…”) in the first place.

This interval can be thought of as containing plausible values of the parameter.

A parameter value is considered to be plausible if it does not produce a small (two-sided) p-value when tested in a null hypothesis.

With this approach, the confidence level of an interval is equal to 1 minus the significance level used for determining rejection/plausibility.

Using a larger confidence level produces a wider interval of plausible values.

Confidence intervals and tests of significance generally give complementary information. That is, null hypothesis values that can be rejected will not be in the confidence interval and vice versa.

They outline the (naive?) approach that I expected to work. Divide the range of values for the unknown parameter p into a grid. For each p, use a permutation test for the sharp null hypothesis. The (approximate) 95% confidence interval for p is all those tested values of p which do not reject the sharp null hypothesis using the 5% significance level.

I realize that I could use a bootstrap approach. But, in an intro course, I would prefer to avoid that complication.

Summary: Can we calculate a confidence interval using a permutation testing approach?

Answer 1

Thanks for your great questions David.

A few thoughts on this:

1. In section 3.1 of our book we are evaluating a single population proportion using a simulated binomial distribution. We ‘invert’ the test here to generate the CI. This should be consistent/accurate because it approximates using the exact binomial distribution to get the CI. Depending on how you define ‘permutation test’ this may be considered a permutation test (simulation) or not (no permutation/shuffling of a variable).

2. We think that a confidence interval for a single quantitative variable (e.g., a single population mean) is a natural place for a bootstrap, however, the easiest bootstrap for a single mean may not be very good vs. an asymptotic approach and so we prefer the 2*SD approach (see next point for more details) to keep things simpler for students.

3. Later in our course, when we use permutation tests that involve shuffling one of two variables to test the hypothesis of no association, we utilize an (approximate) confidence interval with students that uses 2*SD of the null (permutation) distribution. We make it clear that this is an approximately correct interval. We use this for pedagogical reasons, believing that, for a first course in statistics for students who haven’t taken calculus, the various different approaches to get more precise CIs (various bootstraps, +/-4, and various other asymptotic approaches) often leave students ‘bogged down’ in details, and failing to see the ‘big picture’ of a confidence interval as a range of plausible values for a population parameter. It’s the big picture of statistic vs. parameter, and the big picture of using statistic +/- 2*SD of statistic to generate a range of plausible values for the unknown parameter (e.g., 95% confidence interval) that we want to keep students focused on vs. bogged down in lots of technical details and missing the big picture.

You can read more of our thoughts (along with Tim H and others) on our blog: https://www.causeweb.org/sbi/?page_id=534

Answer 2

Often you can use permutation tests to obtain confidence intervals with appropriate coverage, under particular assumptions.

For example, consider the signed rank test (which is a permutation test based on replacing the original absolute differences with their ranks).

• If (as is usually done), you assume a shift-alternative (along with the typical assumptions like independence and so forth), then you can produce a confidence interval for that shift; it is the set of sample-shifts (shifts of paired-differences) that would not be rejected by the test.

• If you instead keep to some different class of alternatives, you may be able to produce a confidence interval for the parameter in that situation; I gave an example recently.

Similarly you can produce intervals for (say) a location shift in a Wilcoxon-Mann-Whitney test (another permutation test).

[You can't produce an interval in the most general case because these tests typically work for many classes of alternatives.]

Similarly, we could have a confidence interval for a population difference in means from a permutation test under a shift alternative (e.g. see John and Robinson, 1983 [1]).

I presume the intent of the article is that you can't generate a confidence interval from a permutation test of means in a more general circumstance.

[1]: R.D. John & J. Robinson (1983) Significance levels and confidence intervals for permutation tests, Journal of Statistical Computation and Simulation, 16:3-4, 161-173

Answer 3

A few additional thoughts: (1) I would argue that +/-2SD where SD is from the null(permutation) distribution is a ‘permutation-based approach’, albeit not exactly what Glen_b points out below (2) As Glen_b notes below you can do things differently for a location shift, but our argument is that (while it is quite straightforward) it is still ‘one more approach’ that students need to learn. Furthermore, while the explicit permutation approach might be fine/easy enough for comparing two groups (means), we wanted the ‘same approach’ to work for regression and two proportions as well and believe that the 2SD approach provided generally works well enough in those contexts as well (3) there is also the question of pool first or shift first is worth considering (4) our focus is on concept, which we think is best emphasized with our approach, but if (in reality) the software is doing more complex computations, then we hope they can conceptually understand what a CI is (and isn't doing) and still interpret/use it correctly.