Tim Hesterberg's "What Teachers Should Know about the Bootstrap" is excellent. On page 77 he writes: "We cannot use permutation testing to obtain confidence intervals."
Is that correct? If so, can someone explain and give a citation.
Introduction to Statistical Investigations by Tintle et al (also excellent) suggests that we can use permutation tests in this way:
SECTION 3.1 Summary
Once a sample proportion has been determined to differ significantly from a hypothesized value for the long-run process probability [using a permutation test], a typical next step is to estimate the value of the long-run process probability with an interval of values, the confidence interval for the parameter. Or our research question may be phrased such that a confidence interval is the more appropriate tool (e.g., “what is the probability that…”) in the first place.
This interval can be thought of as containing plausible values of the parameter.
A parameter value is considered to be plausible if it does not produce a small (two-sided) p-value when tested in a null hypothesis.
With this approach, the confidence level of an interval is equal to 1 minus the significance level used for determining rejection/plausibility.
Using a larger confidence level produces a wider interval of plausible values.
Confidence intervals and tests of significance generally give complementary information. That is, null hypothesis values that can be rejected will not be in the confidence interval and vice versa.
They outline the (naive?) approach that I expected to work. Divide the range of values for the unknown parameter p into a grid. For each p, use a permutation test for the sharp null hypothesis. The (approximate) 95% confidence interval for p is all those tested values of p which do not reject the sharp null hypothesis using the 5% significance level.
I realize that I could use a bootstrap approach. But, in an intro course, I would prefer to avoid that complication.
Summary: Can we calculate a confidence interval using a permutation testing approach?