# Calculating variance between groups in ANOVA

I am extremely confused about getting the between group variance component from an ANOVA. To my understanding, variance within groups ($$V_w$$) equals the mean of squares within groups ($$MSW$$) and the variance between groups ($$V_b$$) can be calculated from the mean of squares between groups ($$MSB$$) as following $$MSB = MSW + k*V_b$$, where $$k$$ is the number of observations per level of the grouping variable.

However, trying to follow this logic by hand or in R I am not able to get the variances right. Let's consider the following example with a subset of mtcars with $$k=4$$.

m4 = mtcars %>% dplyr::filter(cyl == 4) %>% head(n=4)
m6 = mtcars %>% dplyr::filter(cyl == 6) %>% head(n=4)
m8 = mtcars %>% dplyr::filter(cyl == 8) %>% head(n=4)
df = dplyr::bind_rows(m4, m6, m8)


In the dataframe above, the total variance of mpg (var(df$mpg)) is $$21.95$$. If I use ANOVA now, the variances I calculate are: aov(data = df, mpg ~ factor(cyl)) %>% summary() # Df Sum Sq Mean Sq F value Pr(>F) # factor(cyl) 2 160.86 80.43 8.984 0.00717 ** # Residuals 9 80.57 8.95 # --- # Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  $$V_w = 8.95$$, and $$V_b = (80.43-8.95)/4 = 17.87$$, consequently, the total variance estimate is $$Vw + Vb = 26.82$$. But this value is larger than the one above. Why the difference? am I getting something wrong? Also a couple of questions in case anybody knows. If I had different number of observations per level (e.g. as in the original mtcars data set, what would be the "$$k$$" parameter multiplying $$V_b$$ in $$MSB$$ ? Would it be possible to estimate $$V_b$$ in a similar way for a non-parametric design using Kruskal-Wallis? Thanks • I find it striking that$26.82 / 21.95 = 11/9.$– whuber Nov 30, 2021 at 22:37 • Thanks for the pointer I guess? but still confusing. If in the example above (Vw + Vb)*(9/11) = 21.95, which is the total variance. But if one repeats the example with k = 3 and changing the quotient of degrees of freedom to 6/8, it does not work. I'd appreciate a source where I could read how to compute$V_b\$, if what you are suggesting is that the above expression for MSB is wrong. Nov 30, 2021 at 22:58

Here is an approach to a balanced one-factor ANOVA that may have mnemonic value. Consider the model $$Y_{ij} = \mu_i + e_{ij},$$ where $$e_{ij} \stackrel{iid}{\sim}\mathsf{Norm}(0,\sigma).$$

Suppose we have $$h = 3$$ groups and $$k=10$$ replications for each group then DF.Fact $$= h-1 = 2$$ and DF.Resi $$=h(k-1)=3(9) = 27.$$

One says that there are two estimators of the common variance $$\sigma^2:$$ One is the "pooled" variance $$S_w^2$$ of the three groups: $$S_w^2 = (S_1^2 + S_2^2 + S_3^2)/3,$$ in the balanced case. It is a good estimator of $$\sigma^2.$$ whether or not the three groups have the same population mean.

A second estimator $$S_b^2$$ of $$\sigma^2$$ is unbiased only if all three group means are equal to $$\mu,$$ otherwise it tends to be too large. Thus if all group population means are $$\mu,$$ we have $$E(\bar X_1) = E(\bar X_2) = E(\bar X_3) = \mu.$$ Then also $$Var(\bar X_1) = Var(\bar X_2) = Var(\bar X_3) = \sigma^2/k.$$ And the sample variance $$S_{\bar X_i}^2$$ of the three group means estimates $$\sigma^2/k.$$ Then $$S_b^2 = kS_{\bar X_i}^2$$ estimates $$\sigma^2.$$

The F-statistic is MS.Factor/MS.Resi $$S_b^2/ S_w^2.$$ Large values of the F-statistic lead to rejection of the null hypothesis that all group means are equal.

Consider fictitious data for this design and their resulting ANOVA table in R, as follows:

set.seed(1234)
x1 = rnorm(10, 50, 7)
x2 = rnorm(10, 55, 7)
x3 = rnorm(10, 60, 7)
x = c(x1,x2,x3)
g = as.factor(rep(1:3, each=10))
anova(lm(x~g))

Analysis of Variance Table

Response: x
Df  Sum Sq Mean Sq F value   Pr(>F)
g          2  520.01 260.004  6.1835 0.006153 **
Residuals 27 1135.29  42.048
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Now notice that we can get $$S_w^2$$ or MS.Resi = 42.048 of the ANOVA table as follows:

mean(c(var(x1),var(x2),var(x3)))
[1] 42.04788


Also, we can get $$S_b^2$$ or MS.Factor = 260.004 of the ANOVA table as follows:

10*var(c(mean(x1),mean(x2),mean(x3)))
[1] 260.0037


There are similar, but somewhat messier, formulas for $$S_b^2$$ and $$S_w^2$$ if the design is not balanced.