Why is the Mean Square Between (MSB) used in 1-way ANOVA instead of total variance?

Question

I'm having a hard time parsing the logic behind the formulas in the 1-way ANOVA. I'll first establish what I understand so far which will lead into my question.

---

The objective of a 1-way ANOVA is to test the null hypothesis that the population means for all conditions are the same:

$$H_0: \mu_1 = \mu_2 = ... = \mu_k$$

In other words, we are interested in seeing if all the samples are coming from the same population with mean, $\mu$, and variance, $\sigma^2$.

One of the key assumptions of the 1-way ANOVA is the "homogeneity of variance" which states that the variance within each of the populations ($\sigma^2$) is the same. Thus, we can use the pooled variance of all the samples, $S_p$, as an estimator for $\sigma^2$. I believe this pooled variance is more commonly known as the Mean Square Within (MSW):

$$S_p = MSW = \frac{\sum_{j=1}^{k}\sum_{i = 1}^{n_j}(Y_{ij}-\bar{Y_j})^2}{N - k} = \frac{\sum_{j=1}^{k}(n_j - 1)s_j}{N-k} = \frac{SSW}{N-k}$$

---

Ok great, now we have an estimate for the population variance. It would then make sense logically to see how the sample means themselves vary relative to the population variance. If the sample means are all close to each other ("closeness" is of course defined by the estimated population variance), then the null hypothesis is probably true.

But how do we calculate this variance between the sample means? Well here are 2 potential ways:

1.) The "grand variance", S: $$S = \frac{\sum_{j=1}^{k}\sum_{i = 1}^{n_j}(Y_{ij}-\bar{Y})^2}{N - 1} = \frac{TSS}{N-1}$$ where $TSS$ is the Total Sum of Squares.

2.) Using the Mean Square Between (MSB): $$MSB = \frac{\sum_{j=1}^{k}n_j(\bar{Y_j} - \bar{Y})^2}{k - 1} = \frac{SSB}{k-1}$$ where $SSB$ is the Sum of Squares Between.

---

My question: How come the MSB is the preferred way of calculating this "variance between sample means"? Is it an estimator for a certain statistic? Is it because of the useful relationship:

$$SST = SSW + SSB$$

Answer 1

Suppose we have a one-way ANOVA with three levels of the factor, five replications per cell, and common group variance $\sigma^2 = 3^2 = 9.$

To begin, suppose $H_0: \mu_1 = \mu_2 = \mu_3 = 55.$ Then each of the group sample means $\bar X_i,$ for $i = 1,2,3,$ has $Var(\bar X_i) = \sigma^2/n = 9/5,$ so the variance of the three group sample means multiplied by the number $5$ of replications is MSB. Also, the mean of the sample variances (in a balanced design) is MSW.

Consider the following ANOVA, in which data are simulated so that $H_0$ is true. I think the formulas in your question are correct, but it may be more intuitive to look at computations in a simple ANOVA: [Using R.]

set.seed(310)
x1 = rnorm(5, 55, 3)
x2 = rnorm(5, 55, 3)
x3 = rnorm(5, 55, 3)
x = c(x1, x2, x3)
g = as.factor(rep(1:3, each=5))
aov.out = anova(lm(x ~ g));  aov.out

Analysis of Variance Table

Response: x
          Df  Sum Sq Mean Sq F value Pr(>F)
g          2  34.289 17.1445  1.8128 0.2052
Residuals 12 113.492  9.4577  
  

MSB and MSW can be 'captured' from the table as follows:

msb = aov.out[1,3];  msb
[1] 17.14446
msw = aov.out[2,3];  msw
[1] 9.45767

As stated above MSW is the average of the three group variances:

mean(c(var(x1), var(x2), var(x3)))
[1] 9.45767

Also, MSB is the number of replications times the variance of the three group means.

5*var(c(mean(x1), mean(x2), mean(x3)))
[1] 17.14446

Both MSB and MSW are estimates of $\sigma^2 = 9.$ Of course, with so little data, the estimates are not very good. However, it is not difficult to 'wrap' this one example in a 'for' loop and take averages of many similar ANOVAs to show better estimates. [This method of simulation is not fast, elegant, or efficient, but it should be easy to follow.]

set.seed(310)
m = 10^5;  msb = msw = numeric(m)
for(i in 1:m) {
 x1 = rnorm(5, 55, 3)
 x2 = rnorm(5, 55, 3)
 x3 = rnorm(5, 55, 3)
 x = c(x1, x2, x3)
 g = as.factor(rep(1:3, each=5))
 aov.out = anova(lm(x ~ g));  aov.out
 msb[i] = aov.out[1,3];  msb
 msw[i] = aov.out[2,3];  msw
 }
mean(msb);  mean(msw)
[1] 9.058666 # aprx 9
[1] 8.989481 # aprx 9

Now look at a similar ANOVA in which $H_0$ is false because $\mu_1 = 50, \mu_2 = 55, \mu_3 = 60.$

set.seed(2022)
x1 = rnorm(5, 50, 3)  # different
x2 = rnorm(5, 55, 3)  #  group
x3 = rnorm(5, 60, 3)  #   population means
x = c(x1, x2, x3)
g = as.factor(rep(1:3, each=5))
aov.out = anova(lm(x ~ g));  aov.out
    Analysis of Variance Table

Response: x
           Df Sum Sq Mean Sq F value    Pr(>F)    
 g          2 343.56 171.780  16.065 0.0004042 ***
 Residuals 12 128.31  10.692                      
 ---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

msb = aov.out[1,3];  msb
[1] 171.7795
msw = aov.out[2,3];  msw
[1] 10.69249

Now, MSB is much larger. Its expected value is $\sigma^2 + n\theta = 134,$ where $\theta = \frac{1}{3-1}\sum_i (\mu_i - \bar\mu)^2,$ and $\bar\mu = \frac 13 \sum_i \mu_i.$

9 + 5*var(c(50,55,60))
[1] 134

Again here, we have too little data for good estimates of MSB and MSW, but we can come close, using a simulation.

set.seed(2022)
m = 10^5;  msb = msw = numeric(m)
for(i in 1:m) {
 x1 = rnorm(5, 50, 3)  
 x2 = rnorm(5, 55, 3)
 x3 = rnorm(5, 60, 3)
 x = c(x1, x2, x3)
 g = as.factor(rep(1:3, each=5))
 aov.out = anova(lm(x ~ g))
 msb[i] = aov.out[1,3]
 msw[i] = aov.out[2,3]
 }
mean(msb);  mean(msw)
[1] 134.2149
[1] 8.992701