2
$\begingroup$

I conduct 30 independent significance tests on $p<.05$. I observe that in $X=3$ cases the test is rejected. Can this event be caused by chance due to a multiple testing problem?

I understand that $X$ is binomially distributed, with expectation $E(X)=np=1.5$. What is the null hypothesis I should test, is it $H_0: p>.05?$. And how should this test be executed?

EDIT: I think it is $H_0: p<.05$, so we evaluate $P(X>3)=1-P(X <= 3)=.061$. So this is close, but one would not yet reject. From @whuber's comment there seems to be a more sophisticated approach that I need to consider.

$\endgroup$
  • 2
    $\begingroup$ You seem to know the answer to your first question already :-). Consider a more powerful approach: study the distribution of all 30 p-values. Under the null, this will be a sampling distribution from a Uniform$(0,1)$ distribution (equivalently, their negative logarithms will have a $\Gamma(1)$ distribution). It would therefore be interesting to see the histogram of your p-values or--even better--an exponential probability plot. For more about this approach, please consult our posts related to Fisher's method. $\endgroup$ – whuber Apr 11 '14 at 15:57
  • $\begingroup$ @whuber Could you point me to a post that gives hands on guidance on how to combine the p-values using Fisher's method? $\endgroup$ – tomka Apr 11 '14 at 16:07
  • 1
    $\begingroup$ Just follow the link in my comment and read some of the first posts that show up. $\endgroup$ – whuber Apr 11 '14 at 16:49
1
$\begingroup$

There is a whole body of statistical method to deal with Multiplicity. This entails using various types of ANOVAs (depending on your hypothesis testing framework) and numerous related Post Hoc tests run after you have already done the ANOVA. Some of the most commons are Tukey's HSD, Scheffe test, REGWQ test, Dunnet test.

However, you can short cut this whole framework by using well established adjustments to your P value. So, if you want to test for P < 0.05 and you are testing 30 different but related hypothesis, your threshold of significance has to be adjusted by dividing by the number of hypotheses. In your case it would be 0.05/30. That's called the Bonferroni test. There is another similar test that figues some related compounding, and the adustment in this case is: (1 - Confidence Level)^# of hypothesis. In your case: (1 - 95%)^30. This is called the Sidak test.

Note that the two tests derive to very much the same adjustment. They just vary when you go a few decimals out.

Those two tests are very well established, easy to run, and will tell you whether you found out something by chance or not at the 95% Confidence Level by testing multiple hypothesis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.