Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm looking at a problem that has a dependent variable and many independent variables and the probability of getting a type 1 error is 0.05. The questions is: How many tests do you have to run before your chances of getting a statistically significant result are larger than 0.5?
But surely, if I were to run 1 test, my chances of getting a statistically significant result is already 0.95?
I'm so confused..
Assuming the null is true, the chance of an error after $n$ independent tests is
$$ 1 - 0.95^n $$
or one minus the probability of no errors. To figure out how large $n$ needs to be before this exceeds $0.5$ we can look at the following inequality
\begin{align} 1 - 0.95^n &\geq 0.5 \Leftrightarrow \\ 0.95^n &\leq 0.5 \Leftrightarrow \\ n \log(0.95) &\leq \log(0.5) \Leftrightarrow \\ n &\geq \frac{\log(0.5)}{\log(0.95)} . \end{align}
The right hand side is approximately $13.5$ so you need to perform at least $14$ tests.
