Let's say we have 7 samples of data and we wish to test whether there is a significant difference between the means of each sample. We could use t-tests, which would require 21 different t-tests. However, I've read that this isn't advisable as the risk of drawing the wrong conclusion (1 in 20) is increased. Instead, an ANOVA is advised.

But if 21 t-tests were performed using these data and because there are only 7 groups, some of the same data would be used in each test. So although we are calculating 21 different t-tests, we aren't using different data on 21 different occasions. So how does this situation increase the chance of drawing the wrong conclusion?

  • 2
    $\begingroup$ Every time that you do pairwise $t$-tests you have a chance to make a Type I error with probability $\alpha$. Imagine that you read 20 papers and each of them performs one $t$-tests. Statistically, 1 of those tests will yield a false positive, a Type I error. It doesn't matter if you have 20 papers each with one test or one paper with 20 tests. Further, ANOVA doesn't compare each mean with all other means. It tests the hypothesis that all means are equal vs. that at least one mean is different. There are methods to pairwise compare each mean after ANOVA (e.g. Tukey's HSD). $\endgroup$ Jun 1 '13 at 9:16
  • 1
    $\begingroup$ Ok, so it's irrelevant whether the t-tests use the same data, or completely different data. It simply boils down to calculating more t-tests will result in a higher risk of drawing the wrong conclusion. $\endgroup$
    – luciano
    Jun 1 '13 at 9:22
  • $\begingroup$ Just to be pedantic: assuming independence, the probability of a false positive with 20 t-tests under $alpha = 0.05$ is $1-0.95^{20} \approx 0.64$ $\endgroup$
    – abaumann
    Jun 1 '13 at 13:02

Taking a significance level of 5% as given, to simplify the argument:

But if 21 t-tests were performed using these data and because there are only 7 groups, the same data would not be used in each test.

Actually, some tests share some data, but that's beside the point; even if every test were completely independent or every other test, the problem would remain.

So how does this situation increase the chance of drawing the wrong conclusion?

Under the circumstances where $\text{H}_0$ is true, the chance of getting each individual test wrong is still 5%. But the concern being addressed is the overall rate of Type I errors.

Imagine there are no differences between any of the means and each time you do a test, (rather than look at group means and so on) you roll a 20-sided die, which has one side that says "$\bf{R}$" and 19 sides that say "$\bf{A}$" (for "reject" and "don't reject" respectively), giving a Type I error rate of 5% - on any one test, "$\bf{R}$" comes up 5% of the time, and the chance of not getting any "$\bf{R}$" at all is 95%.

But now you roll it 21 times. What is the probability of getting no $R$'s? For independent tests it's $0.95^{21}$ or about $34\%$. The "familywise" Type I error rate is now almost 2/3.

You may not care about the overall rate of Type I errors - and that's perfectly fine; in many circumstances I'd agree with you. But many people do worry about this and reduce their individual-test Type I error rate accordingly.


Say that you conduct $K$ tests to assess $K$ null hypotheses $H_0^{(1)}, \dotsc, H_0^{(K)}$. Further, assume the tests are independent of one another. (As pointed out by @Glen_b, some of your t-tests are not independent... however the idea remains the same.)

Each single test has a type I error probability $\alpha$ associated with it: $$ \Pr[\textrm{reject } H_0^{(k)} \,|\, H_0^{(k)}] = \alpha $$

Let $X$ be the numer of type I errors, i.e. the number of times a null hypothesis is incorrectly rejected.

The overall type I error rate is

\begin{align*} \Pr & [X \geq 1 \,|\, H_0^{(1)}, \dotsc, H_0^{(K)}] \\ & = 1 - \Pr[X = 0 \,|\, H_0^{(1)}, \dotsc, H_0^{(K)}] \\ & = 1 - \prod_{k=1}^K \Pr[\textrm{do not reject } H_0^{(k)} \,|\, H_0^{(k)}] \\ & = 1 - \prod_{k=1}^K \left( 1 - \Pr[\textrm{reject } H_0^{(k)} \,|\, H_0^{(k)}] \right) \\ & = 1 - \prod_{k=1}^K \left(1 - \alpha\right) \\ & = 1 - (1 - \alpha)^K \end{align*}

That is, even though each single test has a type I error rate of $\alpha$, the overall error rate is larger than $\alpha$. enter image description here

It is more complicated to get an explicit formula if the independence assumption is not met. However, it is still the case that the overall rate is larger than $\alpha$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.