I'm looking at a problem that has a dependent variable and many independent variables and the probability of getting a type 1 error is 0.05. The questions is: How many tests do you have to run before your chances of getting a statistically significant result are larger than 0.5?

But surely, if I were to run 1 test, my chances of getting a statistically significant result is already 0.95?

I'm so confused..

  • 2
    $\begingroup$ Do you mean 'your chances of getting at least one ... are larger than 0.5'? You might also like to clarify that you are assuming that there is no true effect of the predictor variables. $\endgroup$
    – mdewey
    Mar 15 '16 at 17:16

Assuming the null is true, the chance of an error after $n$ independent tests is

$$ 1 - 0.95^n $$

or one minus the probability of no errors. To figure out how large $n$ needs to be before this exceeds $0.5$ we can look at the following inequality

\begin{align} 1 - 0.95^n &\geq 0.5 \Leftrightarrow \\ 0.95^n &\leq 0.5 \Leftrightarrow \\ n \log(0.95) &\leq \log(0.5) \Leftrightarrow \\ n &\geq \frac{\log(0.5)}{\log(0.95)} . \end{align}

The right hand side is approximately $13.5$ so you need to perform at least $14$ tests.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.