5
$\begingroup$

Suppose that I have two groups of data of size $50$ and $51$. Assuming that I did an approximate Monte Carlo permutation test with 10,000 permutations randomly drawn, is there a way to find a confidence interval for the estimate of the $p$-value?

Suppose my estimate of the $p$-value is $0.0003$. What would I need to construct such a confidence interval?

Improve this question
$\endgroup$
12
  • 3
    $\begingroup$ "Confidence interval for a p-value" is a contradiction in terms. Confidence intervals apply to parameter estimates, but a p-value most decidedly does not estimate any parameter. One proof of this is to observe that a p-value depends on sample size, whereas a parameter's value does not. Are you possibly asking for a confidence interval on the Monte Carlo estimate of a p-value? $\endgroup$
    – whuber ♦
    Commented Sep 29, 2015 at 20:57
  • 3
    $\begingroup$ Your estimate $p=0.0003$ presumably means that you got 3 permutations out of 10000 yielding statistic at least as extreme as the empirically observed one. So you can ask for a binomial proportion confidence interval given 3 out of 10000 successes. There are several methods to compute one (see wikipedia), in Matlab with binofit(3,10000) I get $(0.00006, 0.0009)$, this is I believe based on the Clopper-Pearson method. $\endgroup$
    – amoeba
    Commented Sep 29, 2015 at 21:53
  • 1
    $\begingroup$ Or repeat the permutation test, say, 10,000 times. Then compute order statistics for both tails (for a 95% interval, look at the 250th smallest and the 250th largest values) and there's your estimated interval. $\endgroup$
    – soakley
    Commented Sep 29, 2015 at 22:03
  • 1
    $\begingroup$ @soakley, if you repeat permutation test with 10000 permutations 10000 times, you did 100 million permutations. Then you can directly estimate your p-value from these 100 million runs and get back to the same question about confidence interval as before. Also, 100 million permutations are rarely feasible. $\endgroup$
    – amoeba
    Commented Sep 29, 2015 at 22:10
  • 1
    $\begingroup$ @whuber I think there is a parameter (of sorts) here -- just not of the original problem (&one conditional on the observed values, as is the case with permutation tests). The parameter is the proportion of permuted samples that would be at least as extreme as the observed sample (as ordered by the test statistic and what is extreme can be identified by the alternative). With a randomization test, we're in a binomial sampling situation, where $n$ is the number of resamples taken and the parameter $p$ is the p-value we would have got if we'd found all the permutations rather than sampled them... $\endgroup$
    – Glen_b
    Commented Sep 30, 2015 at 2:36

0

Reset to default