2
$\begingroup$

I was under the expression that the mean square error (MS) of a 1-way anova table is the estimated variance, and that the between and within group variance add to the total variance. However, using an example I get an result which puts me into doubts. Here the example:

I have a dataset in matlab.

data = [1.5377    0.6923    1.6501    3.7950    5.6715
    2.8339    1.5664    6.0349    3.8759    3.7925
   -1.2588    2.3426    3.7254    5.4897    5.7172
    1.8622    5.5784    2.9369    5.4090    6.6302
    1.3188    4.7694    3.7147    5.4172    5.4889];

If I use the anova1 function I obtain the anova table enter image description here

From this I expected that the variance of the columns is 13.4309 and the variance of the random error is 2.2204. I also expected to obtain that the two MS values add to the total variance. However, if I use the variance function, I get the result

var(data(:)) 
>> 4.0888

This result differens significantly from the MS sum.

I know that

  1. the sum-of-squares add up to the total sum-of-square (first column in picture) $$SS_{total} = SS_{columns} + SS_{error}$$
  2. that the total variance is given by $$var_{total} = SS_{total}/df_{total}$$

However, I often read the following: "we are divide the total variance into a bewteen and a within group variance". I expected that this is actually true. However, from the above example and the two pieces of knowledge I conclude that it's actually the total sum-of-squares, which is seperated and not the total variance. Is this true or is there a way to seperate the total variance into two groups?

$\endgroup$
1
  • 2
    $\begingroup$ It seems that the original question may have been different from its edited version. It is true that sum of squares (not variance) is partitioned. It is also true that the Total sum of squares is the variance of the data without regard for groups. // Both statements can provide useful insights into the computation of the ANOVA table. $\endgroup$
    – BruceET
    May 17 at 18:16

2 Answers 2

5
$\begingroup$

In the example we have $c = 5$ different groups (=columns) and each group consists of $n=5$ different data points. The $MS_{column}$ is given by $MS_{column} = n \cdot \sigma^2_{between}$. Thus, we have to divide $MS_{column}$ by the sample size per group to obtain the "between group variance".

Here the R code:

data.val = c(1.5377, 0.6923, 1.6501, 3.7950, 5.6715, 
             2.8339, 1.5664, 6.0349, 3.8759, 3.7925,
             -1.2588, 2.3426, 3.7254, 5.4897, 5.7172,
             1.8622, 5.5784, 2.9369, 5.4090, 6.6302,
             1.3188, 4.7694, 3.7147, 5.4172, 5.4889)
data.grp = c(rep(1:5, 5))

df = data.frame(val = data.val,
                grp = factor(data.grp) )

library(ggplot2)
gg = ggplot(df, aes(x = grp, y = val, fill=grp)) + 
    geom_boxplot(alpha=0.5) +
    xlab("Columns") +
    ylab("Values")
print(gg)

aov.out   = aov(val ~ grp, df)
anova.out = anova(aov.out)
print(anova.out)


## 
# Calc variance within group
nGrp        = length(unique(data.grp)) # different groups
var.within  = vector(mode = "numeric", length = nGrp)
mean.within = vector(mode = "numeric", length = nGrp)
for ( i in 1:nGrp ){
    idx              = data.grp == i
    var.within[[i]]  = var(data.val[idx])
    mean.within[[i]] = mean(data.val[idx])
}
# The average within group variance is equal to the "Residuals" Mean Sq in the anova table:
# This is Var_within = E[ Var[grp1], Var[grp2], ..., Var[grp5] ]
var.withinAverage = mean(var.within)
print(paste0("var.within = ", var.withinAverage))

# The variance between groups is given by
# Var_between = Var[ E[grp1],  E[grp2], ... E[grp5] ]
var.between = var(mean.within)
print(paste0("var.between = ", var.between))

##
# To obtain the MS_between term, we have to multiply by the sample size per group:
kSample    = sum(data.grp == 1) # number of elements within each group
MS.between = kSample * var.between
print(paste0("MS.between = kSample * var.between = ", MS.between))

##
# Thus, if we invert the last expression we get:
# var.between = MS.between / kSample

Which yields the following output:

Response: val
          Df Sum Sq Mean Sq F value   Pr(>F)   
grp        4 53.723 13.4307  6.0488 0.002332 **
Residuals 20 44.408  2.2204                    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
[1] "var.within = 2.2203876922"
[1] "var.between = 2.6861421478"
[1] "MS.between = kSample * var.between = 13.430710739"

The example above demonstrates

  1. that the anova table is constructed by decomposing the sum-of-squares (and not the variance) into two components,
  2. that the different elements in the anova table are not the within and between group variance components, and
  3. how the anova table is constructed and how it is related to the within and between group variance.

Nevertheless, it is possible to decompose the total variance into a within group variance component and a between group variance component. However, these terms are defined differently. If you like to learn more about this decomposition you will have to read about variance components.

$\endgroup$
1
  • 1
    $\begingroup$ @BruceET: The between group variance is approx. 2.68 and the within group variance is approx. 2.22. I believe the question is, why these two do not add up to the total variance, which is approx. 4.08. I have no clear answer except to agree with Jonas who said: "We do not separate the variances, but the sum-of-squares". Do you agree with this statement? $\endgroup$
    – Semoi
    May 17 at 17:44
3
$\begingroup$

Using R as a calculator, the variance of the 25 observations (without regard to groups) is $4.088775.$

x = c(1.5377,    0.6923,    1.6501,    3.7950,    5.6715, 
      2.8339,    1.5664,    6.0349,    3.8759,    3.7925,
     -1.2588,    2.3426,    3.7254,    5.4897,    5.7172,
      1.8622,    5.5784,    2.9369,    5.4090,    6.6302,
      1.3188,    4.7694,    3.7147,    5.4172,    5.4889)
var(x)
[1] 4.088775

Also, SS(Total)/DF(Total) $= 4.088792,$ which is the same as above, except for slight rounding of SS(Total) = SS(Group) + SS(Resid) in your table.

53.723 + 44.408
[1] 98.131
98.131/24
[1] 4.088792
$\endgroup$
1
  • 1
    $\begingroup$ @Semoi. I attempted to answer the original question, before editing., which I interpreted differently. However, the partition is indeed of the sum of squares, not of "variance. (+1) for your answer which states a correct fact. $\endgroup$
    – BruceET
    May 17 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.