0
$\begingroup$

I have a dataset with an outcome and a big number of independent variables. For each of the independent variables, i would like to measure the effect of the variable on the outcome and test the null hypothesis. The test is designed so that the probability of type-I error is .05. Each of the independent variables is statistically independent of the outcome and all other independent variables.

  1. Suppose we tested the effect of each of 100 variables on the outcome. What is the expected number of times that we would reject the null hypothesis?

  2. Suppose we ran 3 tests. What is the probability of rejecting the null hypothesis (i.e. getting a statistically significant effect) in at least one of the tests?

  3. Suppose we ran 100 tests. What is the probability of rejecting the null hypothesis (i.e. getting a statistically significant effect) in at least one of the tests?

  4. How many tests do we have to run before your chances of getting a statistically significant result are larger than .5?

$\endgroup$
  • $\begingroup$ This looks like a typical textbook or homework assignment. If this is the case, please add the [self-study] tag and read its wiki as we treat those kinds of questions a little differently here. $\endgroup$ – Andy Mar 13 '16 at 16:57
  • 1
    $\begingroup$ Thanks Andy, I'm new here so I don't know the set-up. I'll add the tag now $\endgroup$ – DodgyDavid Mar 13 '16 at 17:44
1
$\begingroup$
  1. With each variable having a probability of false positive of 0.05, the expected value is 0.05 * number of variables = 0.05 * 100 = 5

  2. For "at least" situations, it is generally easier to find the "other cases" and subtract from one. The probability of zero null rejections is the probability of a null NOT rejected raised to the power of the number of experiments = 0.95 ^ 3 = 0.8574. The odds of at least 1 is thus 1 - 0.8574 = 0.1426 (probably no rejections)

  3. 1 - 0.95^100 = 0.9941 (almost surely one or more will be rejected)

  4. 1 - 0.95^n >= 0.5 This was actually just solved about an hour ago at Probability and statistical significance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.