1
$\begingroup$

I'm looking at a problem that has a dependent variable and many independent variables and the probability of getting a type 1 error is 0.05. The questions is: How many tests do you have to run before your chances of getting a statistically significant result are larger than 0.5?

But surely, if I were to run 1 test, my chances of getting a statistically significant result is already 0.95?

I'm so confused..

$\endgroup$
1
  • 2
    $\begingroup$ Do you mean 'your chances of getting at least one ... are larger than 0.5'? You might also like to clarify that you are assuming that there is no true effect of the predictor variables. $\endgroup$
    – mdewey
    Mar 15, 2016 at 17:16

1 Answer 1

5
$\begingroup$

Assuming the null is true, the chance of an error after $n$ independent tests is

$$ 1 - 0.95^n $$

or one minus the probability of no errors. To figure out how large $n$ needs to be before this exceeds $0.5$ we can look at the following inequality

\begin{align} 1 - 0.95^n &\geq 0.5 \Leftrightarrow \\ 0.95^n &\leq 0.5 \Leftrightarrow \\ n \log(0.95) &\leq \log(0.5) \Leftrightarrow \\ n &\geq \frac{\log(0.5)}{\log(0.95)} . \end{align}

The right hand side is approximately $13.5$ so you need to perform at least $14$ tests.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.