# Probability and statistical significance

I'm looking at a problem that has a dependent variable and many independent variables and the probability of getting a type 1 error is 0.05. The questions is: How many tests do you have to run before your chances of getting a statistically significant result are larger than 0.5?

But surely, if I were to run 1 test, my chances of getting a statistically significant result is already 0.95?

I'm so confused..

• Do you mean 'your chances of getting at least one ... are larger than 0.5'? You might also like to clarify that you are assuming that there is no true effect of the predictor variables. Mar 15, 2016 at 17:16

## 1 Answer

Assuming the null is true, the chance of an error after $n$ independent tests is

$$1 - 0.95^n$$

or one minus the probability of no errors. To figure out how large $n$ needs to be before this exceeds $0.5$ we can look at the following inequality

\begin{align} 1 - 0.95^n &\geq 0.5 \Leftrightarrow \\ 0.95^n &\leq 0.5 \Leftrightarrow \\ n \log(0.95) &\leq \log(0.5) \Leftrightarrow \\ n &\geq \frac{\log(0.5)}{\log(0.95)} . \end{align}

The right hand side is approximately $13.5$ so you need to perform at least $14$ tests.