# If the omnibus F-test for testing is not significant, is it still possible for the pairwise comparison for testing to be significant?

Continuous outcome data are collected for three independent groups of equal size (i.e., all group n’s are equal). A linear regression analysis is performed to model the group means. More specifically, indicator variables are created for each of the three groups, and two of the three indicator variables are included in a regression model. Alternatively, all three indicator variables could be included in a no- intercept model. Assume group 1 has the lowest observed mean and group 3 has the highest observed mean.

Design and carry out a simulation experiment to answer A and B below: A) What value for the observed mean for group2 would result in the minimal value for the omnibus F-test for testing: H0: μ1 = μ2 = μ3? That is, would the F statistic be smallest if ... • the observed mean for group 2 is exactly equal to either the observed mean for group 1 or the observed mean for group 3? • the observed mean for group 2 is exactly equal to the average of the observed means for group 1 and group 3? or • the observed mean for group 2 is some other value (but remember it must lie between the observed means for group 1 and group 3)? B) What value for the observed mean for group2 would result in the maximal value for the omnibus F-test for testing: H0: μ1 = μ2 = μ3? C) If the omnibus F-test for testing: H0: μ1 = μ2 = μ3 is not significant, is it still possible for the pairwise comparison for testing H0: μ1 = μ3 to be significant? Justify your answer mathematically (i.e. not with simulation). D) Justify your answer to (A) or (B) mathematically.

I am most confused with the part C for the mathematical solution, and also for A and B, my simulation somehow give me the different answer compare with the theoretical form as below. theoretically seems like the mean of group 1 and 3 make the F statistic the smallest, but simulation show when group2 equal the group1 give the F statistics the smallest value. This problem have been bother me for quite a long time without finding the satisfied solutions, any help would be much appreciated!

set.seed(1)
res <- NULL

for(i in 1:1000) {

#Case i
group1 <- rnorm(50, 5, 1)
group2 <- rnorm(50, 5, 1)
group3 <- rnorm(50, 15, 1)

group <- rep(1:3, each = 50)
y <- c(group1, group2, group3)
mod <- anova(lm(y ~ factor(group)+0))
f1 <- mod$F value[1] #case ii group1 <- rnorm(50, 5, 1) group2 <- rnorm(50, 7, 1) group3 <- rnorm(50, 15, 1) group <- rep(1:3, each = 50) y <- c(group1, group2, group3) mod <- anova(lm(y ~ factor(group)+0)) f2 <- mod$F value[1]

#case iii
group1 <- rnorm(50, 5, 1)
group2 <- rnorm(50, 10, 1)
group3 <- rnorm(50, 15, 1)

group <- rep(1:3, each = 50)
y <- c(group1, group2, group3)
mod <- anova(lm(y ~ factor(group)+0))
f3 <- mod$F value[1] #Case iv group1 <- rnorm(50, 5, 1) group2 <- rnorm(50, 15, 1) group3 <- rnorm(50, 15, 1) group <- rep(1:3, each = 50) y <- c(group1, group2, group3) mod <- anova(lm(y ~ factor(group)+0)) f4 <- mod$F value[1]

res <- rbind(res, c(f1, f2, f3, f4))
}

apply(res, 2, mean)

u1<-5
u3<-15
u2<-seq(5,15,by=0.1)
umean<-(20+u2)/3
fstat<-(u1-umean)^2+(u2-umean)^2+(u3-umean)^2
plot(u2,fstat)

• "C) If the omnibus F-test for testing: $H_0: μ_1 = μ_2 = μ_3$ is not significant, is it still possible for the pairwise comparison for testing $H_0: μ_1 = μ_3$ to be significant? ...." YES, especially if the ad hoc tests comparing pairs of levels use a different kind of test statistic. For example, Tukey's HSD compares the diff btw two levels out of $g$ with the expected maximum difference among $g$ levels. // However, in good statistical practice one does not look ad hoc for diff's when F-test not signif. May 21, 2021 at 23:23
• Exactly, what BruceET said. The $F$-test is there to "protect" you against spurious findings, so don't consult the post hoc at all if it is insignificant. May 22, 2021 at 7:33
• As this seems to be based on an assignment that you once had, please add the self-study tag to this question and read about how such questions are handled. The comments you have received thus far represent a standard way to deal with this in practice, but they don't address your question directly. Specifying this question as self-study might get you better hints for an answer.
– EdM
May 22, 2021 at 19:24

For your first set of problems (A and B), your attempt to mimic a classic ANOVA in R is leading you astray. If you look at the details of a mod returned by your code, you will see that the F-test is based on 3 degrees of freedom for the numerator. An F-test with 3 groups, however, should only have 2 degrees of freedom for the numerator.

What's going on?

When you write:

mod <- anova(lm(y ~ factor(group)+0))


you are forcing a linear regression through the origin. That constraint gives you the (incorrect) extra degree of freedom. Think about what that forcing through the origin also does to sums of squares and mean squares, and to the numerator of the F-statistic. In ANOVA, nothing is tied to the origin and only differences of individual groups from the mean of the groups contribute to the numerator of the F-statistic.

R is smart enough to take a standard linear model with a categorical predictor, without forcing through the origin, and provide the corresponding F-test for ANOVA. Try just removing the +0 from your 4 calls to lm() and see what happens.

So your mathematical justification for the results of A and B, implicit in the last 6 lines of your code, is correct. Just convert the code to the corresponding general equations, and use calculus to find the minimum as a function of u2. The lowest ANOVA F statistic with 3 groups is when the middle group is at the mean of the extreme groups, as you suspected.

With respect to problem C, use the results of problems A and B and D. Consider what happens if the middle group is at the mean of the other 2 groups, the overall F test isn't quite significant, but you only want to compare the top against the bottom group.

• Thanks very much, this is very helpful, and I think I understand the concept of problem C, but how do I prove it by mathematical formula? additional note In (C), you should assume that you are testing H0: μ1 = μ3 by testing an appropriate beta coefficient in the linear regression model or by an appropriate CONTRAST statement for the linear regression model. You are NOT performing an independent sample t-test for comparing these two groups. You are NOT using any of the post-hoc comparison methods you might be familiar with (e.g. Tukey, Scheffé; etc.). May 25, 2021 at 14:01
• @sushi without loss of generality, set the lowest mean to 0, the highest to 1, and the third mean to a. Calculate the variance of (0,1,a) as a function of a. Find the minimum of that expression by taking the first derivative with respect to a and setting to 0. You will get a = 1/2.
– EdM
May 25, 2021 at 22:00
• Thanks I think I got this right and with your help, I am able to answer all the questions except the mathematical solution for Part C, which is still an issue for me, the answer is pretty clear (yes), but how do I show it by mathematical formula? May 27, 2021 at 0:00