Let's say we have 7 samples of data and we wish to test whether there is a significant difference between the means of each sample. We could use t-tests, which would require 21 different t-tests. However, I've read that this isn't advisable as the risk of drawing the wrong conclusion (1 in 20) is increased. Instead, an ANOVA is advised.

But if 21 t-tests were performed using these data and because there are only 7 groups, some of the same data would be used in each test. So although we are calculating 21 different t-tests, we aren't using different data on 21 different occasions. So how does this situation increase the chance of drawing the wrong conclusion?

  • 2
    $\begingroup$ Every time that you do pairwise $t$-tests you have a chance to make a Type I error with probability $\alpha$. Imagine that you read 20 papers and each of them performs one $t$-tests. Statistically, 1 of those tests will yield a false positive, a Type I error. It doesn't matter if you have 20 papers each with one test or one paper with 20 tests. Further, ANOVA doesn't compare each mean with all other means. It tests the hypothesis that all means are equal vs. that at least one mean is different. There are methods to pairwise compare each mean after ANOVA (e.g. Tukey's HSD). $\endgroup$ Commented Jun 1, 2013 at 9:16
  • 1
    $\begingroup$ Ok, so it's irrelevant whether the t-tests use the same data, or completely different data. It simply boils down to calculating more t-tests will result in a higher risk of drawing the wrong conclusion. $\endgroup$
    – luciano
    Commented Jun 1, 2013 at 9:22
  • $\begingroup$ Just to be pedantic: assuming independence, the probability of a false positive with 20 t-tests under $alpha = 0.05$ is $1-0.95^{20} \approx 0.64$ $\endgroup$
    – abaumann
    Commented Jun 1, 2013 at 13:02
  • $\begingroup$ @abaumann the probability of false positive is much less. The 7 groups might be independent but the 20 t-tests are not. When the null hypothesis is true, normal distributed distributions with same variance andd means, then the distribution of the largest t-value relates to the studentised range distribution. $\endgroup$ Commented Dec 25, 2021 at 15:24

2 Answers 2


Taking a significance level of 5% as given, to simplify the argument:

But if 21 t-tests were performed using these data and because there are only 7 groups, the same data would not be used in each test.

Actually, some tests share some data, but that's beside the point; even if every test were completely independent of every other test, the problem would remain.

So how does this situation increase the chance of drawing the wrong conclusion?

Under the circumstances where $\text{H}_0$ is true, the chance of getting each individual test wrong is still 5%. But the concern being addressed is the overall rate of Type I errors.

Imagine there are no differences between any of the means and each time you do a test, (rather than look at group means and so on) you roll a 20-sided die, which has one side that says "$\bf{R}$" and 19 sides that say "$\bf{A}$" (for "reject" and "don't reject" respectively), giving a Type I error rate of 5% - on any one test, "$\bf{R}$" comes up 5% of the time, and the chance of not getting any "$\bf{R}$" at all is 95%.

But now you roll it 21 times. What is the probability of getting no $R$'s? For independent tests it's $0.95^{21}$ or about $34\%$. The "familywise" Type I error rate is now almost 2/3.

You may not care about the overall rate of Type I errors - and that's perfectly fine; in many circumstances I'd agree with you. But many people do worry about this and reduce their individual-test Type I error rate accordingly.


Say that you conduct $K$ tests to assess $K$ null hypotheses $H_0^{(1)}, \dotsc, H_0^{(K)}$. Further, assume the tests are independent of one another. (As pointed out by @Glen_b, some of your t-tests are not independent... however the idea remains the same.)

Each single test has a type I error probability $\alpha$ associated with it: $$ \Pr[\textrm{reject } H_0^{(k)} \,|\, H_0^{(k)}] = \alpha $$

Let $X$ be the numer of type I errors, i.e. the number of times a null hypothesis is incorrectly rejected.

The overall type I error rate is

\begin{align*} \Pr & [X \geq 1 \,|\, H_0^{(1)}, \dotsc, H_0^{(K)}] \\ & = 1 - \Pr[X = 0 \,|\, H_0^{(1)}, \dotsc, H_0^{(K)}] \\ & = 1 - \prod_{k=1}^K \Pr[\textrm{do not reject } H_0^{(k)} \,|\, H_0^{(k)}] \\ & = 1 - \prod_{k=1}^K \left( 1 - \Pr[\textrm{reject } H_0^{(k)} \,|\, H_0^{(k)}] \right) \\ & = 1 - \prod_{k=1}^K \left(1 - \alpha\right) \\ & = 1 - (1 - \alpha)^K \end{align*}

That is, even though each single test has a type I error rate of $\alpha$, the overall error rate is larger than $\alpha$. enter image description here

It is more complicated to get an explicit formula if the independence assumption is not met. However, it is still the case that the overall rate is larger than $\alpha$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.