3
$\begingroup$

In statistics class, we learnt that alpha = type 1 error rate, and type 1 error rate is the probability of wrongly rejecting our null hypothesis when it is true. It is equal to the red area of the following figure.enter image description here

However, very rarely do we have just ONE comparison in a study. For example, in a typical psychology study, we often produce something like this: enter image description here

I counted for you, there are 25 ANOVAs and 250 post hoc independent sample t-tests there... And as indicated by the researchers, they consider the p-value 0.05 to be significant. And if the researchers had used Bonferroni correction, I am afraid nothing would have been considered statistically significant...

Therefore, I want to know can I (or should I) calculate the overall Type 1 Error rate for a STUDY, but not just for a single test? Also, I want to know whether other methods to control for Type 1 Error rate exist.

Thank you very much.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I'm very happy to see non-statisticians come to this conclusion themselves! (+1) for critical thinking. $\endgroup$
    – Frans Rodenburg
    Oct 12, 2017 at 14:24

2 Answers 2

Reset to default
1
$\begingroup$

The Wikipedia article here https://en.wikipedia.org/wiki/Multiple_comparisons_problem has more detail and several references. Basically the Bonferroni is very conservative and there are several other options that are not as conservative.

Here are a couple of articles (medical area instead of psychology, but the principles still hold):

http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(05)66461-6/fulltext

http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(05)66516-6/fulltext

These articles give an overview and also discuss some of the cases where it is important to adjust and some where it may not be needed.

Improve this answer
$\endgroup$
1
$\begingroup$

The term you're looking for when you say you want to 'calculate the overall Type 1 Error rate for a STUDY, but not just for a single test' is the 'False Discovery Rate'. There are numerous definitions of the false discovery rate, but Wikipedia is often a good start: https://en.wikipedia.org/wiki/False_discovery_rate.

A false discovery rate will help to control the proportion of discoveries that are false, whereas the Bonferonni Correction will control for the probability of at least one false positive.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.