1
$\begingroup$

So, I have a data about 6 groups. And I made an ANOVA test in R, it gave me F value of 15.34 and a small p value. What could indicate that I should make a pairwise ANOVA test between two of these groups separately? Boxplot? Should I just check whether the boxplot of one of the group stands out and then do the pairwise ANOVA test? I don't want to make pairwise test for all possible combinations of groups, rather, somehow detect which groups should be tested separately?

$\endgroup$
1
  • $\begingroup$ There are very many ways to do post hoc analysis of six levels of a one-way ANOVA. I suggest one commonly used method in my Answer. $\endgroup$
    – BruceET
    May 12, 2020 at 22:12

2 Answers 2

1
$\begingroup$

Below are data with samples from six populations A-F. Using a one-way ANOVA, we see that not all six populations have the same mean.

set.seed(512)
a = rnorm(10, 100, 15);  b = rnorm(10, 100, 15)
c = rnorm(10, 100, 15);  d = rnorm(10, 120, 15)
e = rnorm(10, 140, 15);  f = rnorm(10, 140, 15)
x = c(a,b,c,d,e,f)
g = as.factor(rep(1:6, each=10))

Specifically, a one-way ANOVA shows significant differences among group means, with a P-value near 0.

aov.out = aov(x ~ g)
summary(aov.out)
            Df Sum Sq Mean Sq F value  Pr(>F)    
g            5  18419    3684   15.37 2.2e-09 ***
Residuals   54  12941     240                    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Now we need to do a post hoc analysis to see which differences among group means are significant. Box plots of the six samples may serve as a guide where to look for significant differences. It seems that groups A, B, and C are much alike, that groups E and F may have larger means, and that D is intermediate. [Knowing how the data were simulated, this would be no surprise. But in a real experiment we would have no way to know exact population means.]

boxplot(x ~ g, vol="skyblue2", pch=20,
        names=c("A","B","C","D","E","F"))

enter image description here

There are ${6 \choose 2} = 15$ possible pairs to look at. If we do fifteen tests at the 5% level, we risk 'false discovery'. There are several ad hoc methods that adjust the level of each comparison so that the 'family' of comparisons has an overall significance rate of 5%. Tukey's HSD method is one of them.

The Tukey procedure does all 15 comparisons, making CIs for each difference. The CIs with endpoints of the same sign indicate the significant differences. Also, you can look for adjusted P-values below 5%. I have put #s at the ends of the key rows.

TukeyHSD(aov.out)
  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = x ~ g)

$g
          diff        lwr      upr     p adj
2-1  0.9084022 -19.545937 21.36274 0.9999942
3-1 -3.7750648 -24.229404 16.67927 0.9939535
4-1 14.7886161  -5.665723 35.24296 0.2848278
5-1 37.6989627  17.244623 58.15330 0.0000187 #
6-1 38.4772758  18.022936 58.93162 0.0000125 #
3-2 -4.6834670 -25.137807 15.77087 0.9838191
4-2 13.8802139  -6.574126 34.33455 0.3531239
5-2 36.7905604  16.336221 57.24490 0.0000299 #
6-2 37.5688736  17.114534 58.02321 0.0000200 #
4-3 18.5636809  -1.890659 39.01802 0.0958102
5-3 41.4740274  21.019688 61.92837 0.0000026 #
6-3 42.2523406  21.798001 62.70668 0.0000017 #
5-4 22.9103465   2.456007 43.36469 0.0196843 #
6-4 23.6886597   3.234320 44.14300 0.0143840 #
6-5  0.7783132 -19.676026 21.23265 0.9999973

It seems that there are no significant differences among A,B,C and not between E & F. But the lowest three and the top two are different from each other. Group D may be intermediate but the Tukey procedure does not show it as significantly different from either ABC or EF. [There is no guarantee that post hoc tests can partition levels of the factor into tidy disjoint clusters of groups with no significant differences among them). Here, the status of D is left unresolved.]

If you wanted to use the boxplots as a guide you might do two-sample t tests between B & E to show 'clusters' ABC and EF differ, and between A & B, E & F, B & D, and E & E to find no additional significant differences. Then you could try to use the Bonferroni method of avoiding false discovery, which adjusts P-values according to the number of comparisons made. Formally, you would have made five, but informally from the boxplot you have made others. So you will find differences of opinion how to use the Bonferroni method here.

You say, "I don't want to make pairwise test for all possible combinations of groups,...." and I understand that. But, using R, they're all made automatically and according to a recognized method of avoiding false discovery. So for a one-way ANOVA with six levels of the factor, I'd use Tukey's HSD method and be done with it.

$\endgroup$
0
$\begingroup$

Ideally, this should be motivated by the scientific question of interest. Why are you interested in pairwise comparisons between any combination of groups? You already know that at least two of your groups are not equal.

Looking for two groups that have a big difference in mean and then testing that without adjusting for multiple comparisons will still inflate your type-I error since you're only testing because you saw a large difference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.