# Type I Error in research: What is the alpha of a study especially when there are multiple comparisons?

In statistics class, we learnt that alpha = type 1 error rate, and type 1 error rate is the probability of wrongly rejecting our null hypothesis when it is true. It is equal to the red area of the following figure.

However, very rarely do we have just ONE comparison in a study. For example, in a typical psychology study, we often produce something like this:

I counted for you, there are 25 ANOVAs and 250 post hoc independent sample t-tests there... And as indicated by the researchers, they consider the p-value 0.05 to be significant. And if the researchers had used Bonferroni correction, I am afraid nothing would have been considered statistically significant...

Therefore, I want to know can I (or should I) calculate the overall Type 1 Error rate for a STUDY, but not just for a single test? Also, I want to know whether other methods to control for Type 1 Error rate exist.

Thank you very much.

• I'm very happy to see non-statisticians come to this conclusion themselves! (+1) for critical thinking. Oct 12, 2017 at 14:24

The Wikipedia article here https://en.wikipedia.org/wiki/Multiple_comparisons_problem has more detail and several references. Basically the Bonferroni is very conservative and there are several other options that are not as conservative.

Here are a couple of articles (medical area instead of psychology, but the principles still hold):

http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(05)66461-6/fulltext

http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(05)66516-6/fulltext

These articles give an overview and also discuss some of the cases where it is important to adjust and some where it may not be needed.

The term you're looking for when you say you want to 'calculate the overall Type 1 Error rate for a STUDY, but not just for a single test' is the 'False Discovery Rate'. There are numerous definitions of the false discovery rate, but Wikipedia is often a good start: https://en.wikipedia.org/wiki/False_discovery_rate.

A false discovery rate will help to control the proportion of discoveries that are false, whereas the Bonferonni Correction will control for the probability of at least one false positive.