# Why is the Mean Square Between (MSB) used in 1-way ANOVA instead of total variance?

I'm having a hard time parsing the logic behind the formulas in the 1-way ANOVA. I'll first establish what I understand so far which will lead into my question.

The objective of a 1-way ANOVA is to test the null hypothesis that the population means for all conditions are the same:

$$H_0: \mu_1 = \mu_2 = ... = \mu_k$$

In other words, we are interested in seeing if all the samples are coming from the same population with mean, $$\mu$$, and variance, $$\sigma^2$$.

One of the key assumptions of the 1-way ANOVA is the "homogeneity of variance" which states that the variance within each of the populations ($$\sigma^2$$) is the same. Thus, we can use the pooled variance of all the samples, $$S_p$$, as an estimator for $$\sigma^2$$. I believe this pooled variance is more commonly known as the Mean Square Within (MSW):

$$S_p = MSW = \frac{\sum_{j=1}^{k}\sum_{i = 1}^{n_j}(Y_{ij}-\bar{Y_j})^2}{N - k} = \frac{\sum_{j=1}^{k}(n_j - 1)s_j}{N-k} = \frac{SSW}{N-k}$$

Ok great, now we have an estimate for the population variance. It would then make sense logically to see how the sample means themselves vary relative to the population variance. If the sample means are all close to each other ("closeness" is of course defined by the estimated population variance), then the null hypothesis is probably true.

But how do we calculate this variance between the sample means? Well here are 2 potential ways:

1.) The "grand variance", S: $$S = \frac{\sum_{j=1}^{k}\sum_{i = 1}^{n_j}(Y_{ij}-\bar{Y})^2}{N - 1} = \frac{TSS}{N-1}$$ where $$TSS$$ is the Total Sum of Squares.

2.) Using the Mean Square Between (MSB): $$MSB = \frac{\sum_{j=1}^{k}n_j(\bar{Y_j} - \bar{Y})^2}{k - 1} = \frac{SSB}{k-1}$$ where $$SSB$$ is the Sum of Squares Between.

My question: How come the MSB is the preferred way of calculating this "variance between sample means"? Is it an estimator for a certain statistic? Is it because of the useful relationship:

$$SST = SSW + SSB$$

• Three things to consider: (1) SSB + SSW = SST. So, if you used MSB and MST to make the variance ratio, you would be comparing the part with the whole, not the two independent estimates of variance with each other. (2) Whether nor not $H_0$ is true, MSW is an unbiased estimator of the common variance $\sigma^2.$ Only if $H_0$ is true, is MSB an unbiased estimator or $\sigma^2,$ otherwise it tends to be too large. (3) MSB/MSW has an F-distribution, and MSB/MST does not. Mar 10, 2022 at 23:47
• @BruceET If the null hypothesis is true, are MSB, MSW, and MST all unbiased estimators of $\sigma^2$? And if not, only MSW is?
– Nova
Mar 11, 2022 at 4:07
• Right. However, MST is not usually used to estimate $\sigma^2.$ Mar 11, 2022 at 6:28

Suppose we have a one-way ANOVA with three levels of the factor, five replications per cell, and common group variance $$\sigma^2 = 3^2 = 9.$$

To begin, suppose $$H_0: \mu_1 = \mu_2 = \mu_3 = 55.$$ Then each of the group sample means $$\bar X_i,$$ for $$i = 1,2,3,$$ has $$Var(\bar X_i) = \sigma^2/n = 9/5,$$ so the variance of the three group sample means multiplied by the number $$5$$ of replications is MSB. Also, the mean of the sample variances (in a balanced design) is MSW.

Consider the following ANOVA, in which data are simulated so that $$H_0$$ is true. I think the formulas in your question are correct, but it may be more intuitive to look at computations in a simple ANOVA: [Using R.]

set.seed(310)
x1 = rnorm(5, 55, 3)
x2 = rnorm(5, 55, 3)
x3 = rnorm(5, 55, 3)
x = c(x1, x2, x3)
g = as.factor(rep(1:3, each=5))
aov.out = anova(lm(x ~ g));  aov.out

Analysis of Variance Table

Response: x
Df  Sum Sq Mean Sq F value Pr(>F)
g          2  34.289 17.1445  1.8128 0.2052
Residuals 12 113.492  9.4577



MSB and MSW can be 'captured' from the table as follows:

msb = aov.out[1,3];  msb
 17.14446
msw = aov.out[2,3];  msw
 9.45767


As stated above MSW is the average of the three group variances:

mean(c(var(x1), var(x2), var(x3)))
 9.45767


Also, MSB is the number of replications times the variance of the three group means.

5*var(c(mean(x1), mean(x2), mean(x3)))
 17.14446


Both MSB and MSW are estimates of $$\sigma^2 = 9.$$ Of course, with so little data, the estimates are not very good. However, it is not difficult to 'wrap' this one example in a 'for' loop and take averages of many similar ANOVAs to show better estimates. [This method of simulation is not fast, elegant, or efficient, but it should be easy to follow.]

set.seed(310)
m = 10^5;  msb = msw = numeric(m)
for(i in 1:m) {
x1 = rnorm(5, 55, 3)
x2 = rnorm(5, 55, 3)
x3 = rnorm(5, 55, 3)
x = c(x1, x2, x3)
g = as.factor(rep(1:3, each=5))
aov.out = anova(lm(x ~ g));  aov.out
msb[i] = aov.out[1,3];  msb
msw[i] = aov.out[2,3];  msw
}
mean(msb);  mean(msw)
 9.058666 # aprx 9
 8.989481 # aprx 9


Now look at a similar ANOVA in which $$H_0$$ is false because $$\mu_1 = 50, \mu_2 = 55, \mu_3 = 60.$$

set.seed(2022)
x1 = rnorm(5, 50, 3)  # different
x2 = rnorm(5, 55, 3)  #  group
x3 = rnorm(5, 60, 3)  #   population means
x = c(x1, x2, x3)
g = as.factor(rep(1:3, each=5))
aov.out = anova(lm(x ~ g));  aov.out
Analysis of Variance Table

Response: x
Df Sum Sq Mean Sq F value    Pr(>F)
g          2 343.56 171.780  16.065 0.0004042 ***
Residuals 12 128.31  10.692
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

msb = aov.out[1,3];  msb
 171.7795
msw = aov.out[2,3];  msw
 10.69249


Now, MSB is much larger. Its expected value is $$\sigma^2 + n\theta = 134,$$ where $$\theta = \frac{1}{3-1}\sum_i (\mu_i - \bar\mu)^2,$$ and $$\bar\mu = \frac 13 \sum_i \mu_i.$$

9 + 5*var(c(50,55,60))
 134


Again here, we have too little data for good estimates of MSB and MSW, but we can come close, using a simulation.

set.seed(2022)
m = 10^5;  msb = msw = numeric(m)
for(i in 1:m) {
x1 = rnorm(5, 50, 3)
x2 = rnorm(5, 55, 3)
x3 = rnorm(5, 60, 3)
x = c(x1, x2, x3)
g = as.factor(rep(1:3, each=5))
aov.out = anova(lm(x ~ g))
msb[i] = aov.out[1,3]
msw[i] = aov.out[2,3]
}
mean(msb);  mean(msw)
 134.2149
 8.992701