I want to perform a statistical hypothesis test, however I don't know the exact distribution of my test statistic under $H_0$. Therefore, I need to calculate a Monte Carlo estimate $\hat{p}$ of the true p-value.

Problem is, that the test statistic is pretty expensive to compute, so I want to stop permutations as early as possible while still knowing whether I can reject $H_0$ with a certain confidence.

There are several methods out there to limit the number of permutations in advance (see here, e.g.), as well as sequential Monte Carlo tests that are supposed to stop even earlier than pre-defined bounds in many cases by iteratively updating the bounds (like the one proposed by Axel Gandy).

On the other hand, if I got it right the CI of $\hat{p}$ can easily be computed using a binomial proportion confidence interval.

My question:
Since it would be sufficient to know whether $\hat{p} \leq \alpha$ for a pre-defined confidence level $\alpha$ (rather than having an exact estimate of $p$), would it be possible to just compute the binomial proportion CI after each iteration and stop simulations as soon as $\alpha$ is outside the CI?

My idea would be that this also stops earlier than pre-defined bounds while being less complex than e.g. Gandy's approach.

And if the above is a valid approach, secondary question is what is the difference to approaches like Gandy (i.e. what do I get for the increase in complexity)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.