0
$\begingroup$

I understand that the minimum number of simulations ($m$) required for a Monte Carlo test at a particular significance level can be determined by: $$\alpha=1/(m+1)$$ for a one-tailed test and $$\alpha=2/(m + 1)$$ for a two tailed test, such that a one-tailed test at a significance level of 5% would required (a minimum of) 19 simulations, a two-tailed test at a significance level of 5% would require (a minimum of) 39 simulations, etc.

My question pertains to whether or not it is theoretically valid to increase the number of simulations beyond the minimum that are required for a particular significance level. As an analyst, I have always worked under the assumption of "the more the merrier", as I figured that more simulations would give you a better estimate of the shape of the sampling distribution under the null hypothesis, but I am really not sure, from the point of view of a statistician, whether this is valid.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

The formulas you posted give you the absolute minimum sample size, since anything less then this would make it impossible to observe the event of interest with probability $\le \alpha$. The same as two coin tosses is absolute minimum to judge, using Monte Carlo simulation, if coin does not have two identical sides, since with one coin toss it would be impossible for you to see the other side of the coin. This doesn't mean that the absolute minimum sample size is anything close to optimal sample size for this purpose. You can check, for example, this thread on bootstrap that suggests sample sizes of at least 1000, with 10,000 being the common suggestion among the answers. You can also check this thread about choosing optimal number of iterations for Monte Carlo simulation.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Indeed, I'd usually regard even 10000 for bootstrap and randomization/permutation tests as a bit small and do more unless it's very slow to evaluate. Consider that if you're looking at $\alpha=0.01$ (even without adjusting for multiple comparisons) you don't have quite two figure accuracy on your p-values. It would be really annoying if your 0.009 p-value came out as 0.011 ... $\endgroup$
    – Glen_b
    Sep 10, 2017 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.