13
$\begingroup$

I'm learning randomization tests right now. There are two questions coming to my mind:

  1. Yes, it's easy and intuitive how p-value is computed with randomization test (which I think is the same as permutation test?). However, how could we also generate a 95% confidence interval as we do with normal parametric tests?

  2. As I'm reading a document from University of Washington on permutation tests, there is a sentence on page 13 that says:

    With 1000 permutations ...., the uncertainty near p = 0.05 is about $\pm 1\% $.

    I wonder how we get this uncertainty.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
15
$\begingroup$

However, how could we also generate a 95% confidence interval as we do with normal parametric tests?

Here's one way you could generate an interval from a resampling test, though it's not always appropriate to consider it a confidence interval$^\dagger$. For a specific example, take a test for a two-sample difference in means. Consider shifting the second sample by $\delta$ (which can be positive or negative). Then the set of $\delta$ values which would lead to non-rejection by the test at level $\alpha$ could be used as a nominally $1-\alpha$ confidence interval for the difference in means.

$\dagger$ Some authors (e.g. [1], p364 et seq, [2]) call an interval constructed this way (parameter values not rejected by the test) a consonance interval -- which is a better name than confidence interval for it (though many people simply ignore the difference; for example I believe Cox & Hinkley call these confidence intervals) because the approach doesn't necessarily give intervals that have the desired coverage (in many situations it's possible to see that it should); the name conveys something about what the interval does tell you (an interval of values consistent with the data).

Gelman includes discussion of why it can sometimes be problematic to universally consider them confidence intervals here.

It's not hard to explore the coverage under particular sets of assumptions (via simulation), though, and there's no lack of people calling bootstrap intervals "confidence intervals" (even when they are sometimes seen to have nothing like the claimed coverage).

More details on how to do it in the two sample difference-in-means case are discussed in [3], where they're called randomization confidence intervals and a claim is made there about when they're exact (which claim I haven't tried to evaluate).

With 1000 permutations ...., the uncertainty near p = 0.05 is about ±1%.

I wonder how we get this uncertainty?

The estimated p-value is a straight binomial proportion. So it has the same standard error as any other binomial proportion, $\sqrt{\frac{p(1-p)}{n}}$.

So if $p = 0.05$ and $n=1000$, the standard error of the observed proportion is about $0.0069$. A $90\%$ CI would be $\pm 1.13\%$ [Alternatively, $\pm 1\%$ is about $1.45$ standard errors each side, which would correspond to a confidence interval for the underlying p-value of a bit over $85\%$]

So at least in a rough sense you could talk about the uncertainty being "about 1%"

--

[1] Kempthorne and Folks (1971),
Probability, Statistics, and data analysis,
Iowa State University Press

[2] LaMotte L.R. and Volaufová J, (1999),
"Prediction Intervals via Consonance Intervals",
Journal of the Royal Statistical Society. Series D (The Statistician), Vol. 48, No. 3, pp. 419-424

[3] Ernst, M.D. (2004),
"Permutation Methods: A Basis for Exact Inference", Statistical Science, Vol. 19, No. 4, 676–685

Improve this answer
$\endgroup$
8
  • $\begingroup$ Re "The estimated p-value is a straight binomial proportion": this is what I thought at first, but note that there are a finite number of possible resamples (especially with small datasets) so you're not necessarily getting independent data from each resample. Some specific permutations may be repeated. $\endgroup$
    – Matt
    Feb 15, 2021 at 18:13
  • $\begingroup$ Imagine I have a standard deck of cards and I want to know the probability of drawing at least one pair of face cards (JJ,QQ,KK) in a deal of four cards. Noting that the number of hands of four cards is finite (i.e. that the target population is finite): (i) I could proceed by enumeration (either implicit or explicit) -- to count how many of the possible draws of four cards satisfy that at least one pair of face cards condition and hence, what proportion of the total number of hands of four cards that is; or $\endgroup$
    – Glen_b
    Feb 15, 2021 at 23:36
  • $\begingroup$ (ii) I could shuffle the deck many times, and each time draw four cards (i.e. sample with replacement), and see what proportion of the total that desired outcome is. In this particular instance (i) is easy to do without complete enumeration - we can identify how many hands there are in total (${52 \choose 4}$) and how many satisfy the condition without explicitly listing all the possible combinations. However, method (ii) works perfectly well for estimating this proportion to any level of accuracy. Yes, if we draw a lot of samples we will see repeats of some of those combinations $\endgroup$
    – Glen_b
    Feb 15, 2021 at 23:37
  • $\begingroup$ (we might see the specific hand "two queens, a nine and a four" come up several times, for example) but this is of no consequence (aside from potentially requiring more samples than we might have been able to achieve if we sampled without replacement). Further, the count of "success" events in this set of redraws is binomial and we can take advantage of that in computing a confidence interval for our proportion. $\endgroup$
    – Glen_b
    Feb 15, 2021 at 23:37
  • $\begingroup$ Thanks for the explanation Glen. I think there are two issues here: $\endgroup$
    – Matt
    Feb 17, 2021 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.