Calculating variance between groups in ANOVA

Question

I am extremely confused about getting the between group variance component from an ANOVA. To my understanding, variance within groups ($V_w$) equals the mean of squares within groups ($MSW$) and the variance between groups ($V_b$) can be calculated from the mean of squares between groups ($MSB$) as following $MSB = MSW + k*V_b$, where $k$ is the number of observations per level of the grouping variable.

However, trying to follow this logic by hand or in R I am not able to get the variances right. Let's consider the following example with a subset of mtcars with $k=4$.

m4 = mtcars %>% dplyr::filter(cyl == 4) %>% head(n=4)
m6 = mtcars %>% dplyr::filter(cyl == 6) %>% head(n=4)
m8 = mtcars %>% dplyr::filter(cyl == 8) %>% head(n=4)
df = dplyr::bind_rows(m4, m6, m8)

In the dataframe above, the total variance of mpg (var(df$mpg)) is $21.95$.

If I use ANOVA now, the variances I calculate are:

aov(data = df, mpg ~ factor(cyl)) %>% summary()
#             Df Sum Sq Mean Sq F value  Pr(>F)   
# factor(cyl)  2 160.86   80.43   8.984 0.00717 **
# Residuals    9  80.57    8.95                   
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

$V_w = 8.95$, and $V_b = (80.43-8.95)/4 = 17.87$, consequently, the total variance estimate is $Vw + Vb = 26.82$. But this value is larger than the one above. Why the difference? am I getting something wrong?

Also a couple of questions in case anybody knows. If I had different number of observations per level (e.g. as in the original mtcars data set, what would be the "$k$" parameter multiplying $V_b$ in $MSB$ ? Would it be possible to estimate $V_b$ in a similar way for a non-parametric design using Kruskal-Wallis?

Thanks

Answer 1

Here is an approach to a balanced one-factor ANOVA that may have mnemonic value. Consider the model $Y_{ij} = \mu_i + e_{ij},$ where $e_{ij} \stackrel{iid}{\sim}\mathsf{Norm}(0,\sigma).$

Suppose we have $h = 3$ groups and $k=10$ replications for each group then DF.Fact $= h-1 = 2$ and DF.Resi $=h(k-1)=3(9) = 27.$

One says that there are two estimators of the common variance $\sigma^2:$ One is the "pooled" variance $S_w^2$ of the three groups: $S_w^2 = (S_1^2 + S_2^2 + S_3^2)/3,$ in the balanced case. It is a good estimator of $\sigma^2.$ whether or not the three groups have the same population mean.

A second estimator $S_b^2$ of $\sigma^2$ is unbiased only if all three group means are equal to $\mu,$ otherwise it tends to be too large. Thus if all group population means are $\mu,$ we have $E(\bar X_1) = E(\bar X_2) = E(\bar X_3) = \mu.$ Then also $Var(\bar X_1) = Var(\bar X_2) = Var(\bar X_3) = \sigma^2/k.$ And the sample variance $S_{\bar X_i}^2$ of the three group means estimates $\sigma^2/k.$ Then $S_b^2 = kS_{\bar X_i}^2$ estimates $\sigma^2.$

The F-statistic is MS.Factor/MS.Resi $S_b^2/ S_w^2.$ Large values of the F-statistic lead to rejection of the null hypothesis that all group means are equal.

Consider fictitious data for this design and their resulting ANOVA table in R, as follows:

set.seed(1234)
x1 = rnorm(10, 50, 7)
x2 = rnorm(10, 55, 7)
x3 = rnorm(10, 60, 7)
x = c(x1,x2,x3)
g = as.factor(rep(1:3, each=10))
anova(lm(x~g))
 
Analysis of Variance Table

Response: x
          Df  Sum Sq Mean Sq F value   Pr(>F)   
g          2  520.01 260.004  6.1835 0.006153 **
Residuals 27 1135.29  42.048                    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Now notice that we can get $S_w^2$ or MS.Resi = 42.048 of the ANOVA table as follows:

mean(c(var(x1),var(x2),var(x3)))
[1] 42.04788

Also, we can get $S_b^2$ or MS.Factor = 260.004 of the ANOVA table as follows:

10*var(c(mean(x1),mean(x2),mean(x3)))
[1] 260.0037

There are similar, but somewhat messier, formulas for $S_b^2$ and $S_w^2$ if the design is not balanced.