Probability and null hypothesis testing with a large number of independent variables

Question

I have a dataset with an outcome and a big number of independent variables. For each of the independent variables, i would like to measure the effect of the variable on the outcome and test the null hypothesis. The test is designed so that the probability of type-I error is .05. Each of the independent variables is statistically independent of the outcome and all other independent variables.

1. Suppose we tested the effect of each of 100 variables on the outcome. What is the expected number of times that we would reject the null hypothesis?

2. Suppose we ran 3 tests. What is the probability of rejecting the null hypothesis (i.e. getting a statistically significant effect) in at least one of the tests?

3. Suppose we ran 100 tests. What is the probability of rejecting the null hypothesis (i.e. getting a statistically significant effect) in at least one of the tests?

4. How many tests do we have to run before your chances of getting a statistically significant result are larger than .5?

Answer 1

1. With each variable having a probability of false positive of $0.05, $ the expected value is $0.05 \times\text{ number of variables} = 0.05 \times 100 = 5.$

2. For "at least" situations, it is generally easier to find the "other cases" and subtract from one. The probability of zero null rejections is the probability of a null NOT rejected raised to the power of the number of experiments $= 0.95 ^ 3 = 0.8574. $ The odds of at least $1$ is thus $1 - 0.8574 = 0.1426$ (probably no rejections).

3. $1 - 0.95^{100} = 0.9941$ (almost surely one or more will be rejected).

4. $1 - 0.95^n \geq 0.5.$ This was actually just solved about an hour ago at Probability and statistical significance