4
$\begingroup$

I perform a permutation test multiple times on different datasets, each time I am only concerned about significant $p$ values. To reduce computation time would it be correct to introduce this kind of stopping rule: After a certain number of $N$ permutations to check whether $p$ is greater than a particular value. So, for example if $p>0.1$ after $N$=200 permutations then the lower bound of 95% confidence interval would be greater than $0.05$. Therefore, calculations could be stopped as a true $p$ is not significant. Just want to make sure I am doing it right. Thank you.

$\endgroup$
  • $\begingroup$ are your permutations properly randomised, if that makes sense in your context? $\endgroup$ – Robert Jones Jun 7 '13 at 11:47
  • $\begingroup$ yes, all permutations are independent. Each time I do permutation of initial data set, I calculate the new value of statistics, and count the number of cases when statistics from permutation is greater or equal than initial statistics. So, $p$ is a proportion of such cases. $\endgroup$ – Math_cat Jun 7 '13 at 12:08
3
$\begingroup$

If it's not quite significant, do you keep simulating?

If so, then no, the significance level is no longer the desired level when you do that. It's substantially affected by this checking for significance along the way.

You can adjust things so that the properties are what you want, but it's not a matter of just stopping early.

One way of doing this is called sequential probability ratio testing (SPRT).

$\endgroup$
  • $\begingroup$ The idea was to stop simulation if the 95% confidence interval of estimated $p$ value is located above 0.05 after particular number of permutations. I am using the normal approximation to obtain the CI in this case. So, no, I don't keep simulating in this case. Thank you for pointing out the SPRT approach. Initially, I performed around 2500 permutations in order to estimate p-value of order of 0.01. $\endgroup$ – Math_cat Jun 7 '13 at 12:05
  • $\begingroup$ If you decide whether to stop at some point along the way (and clearly if you don't stop,you keep going) based on a p-value, you get the problem of looking at the p-value more than once. It's still the same issue $\endgroup$ – Glen_b Jan 27 '17 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.