ANOVA sum of squares between groups

Question

I'm trying to get an intuitive understanding of why the sum of squares between groups needs to be multiplied by the number of observations within each group. Using the iris dataset in R as an example, the sum of squares needs to be multiplied by 50:

meanOverall = mean(iris$Sepal.Length)

meanSetosa = mean(iris$Sepal.Length[iris$Species=='setosa'])

meanVersicolor = mean(iris$Sepal.Length[iris$Species=='versicolor'])

meanVirginica = mean(iris$Sepal.Length[iris$Species=='virginica']) 

sumsquaresBetweenGroups = sum((meanSetosa - meanOverall) * (meanSetosa - meanOverall) + (meanVersicolor - meanOverall) * (meanVersicolor - meanOverall) + 
(meanVirginica - meanOverall) * (meanVirginica - meanOverall)) * 50

Why do you need to multiply the between groups sum of squares by the number of observations in each group?

In addition, what if $n$ is different in each group. What would you do if there were 45, 55 and 60 observations in each of the three groups of Species in the iris dataset?

Answer 1

Why do you need to multiply the between groups sum of squares by the number of observations in each group?

The first bullet in the list below answers the question.

---

The deviation of observation $Y_{ij}$ ($i$ is for the group and $j$ is for the observation within a group) around the overall mean $\bar{Y}_{\cdot\cdot}$ can be written as the sum of two components (subtract and add $\bar{Y}_{i\cdot}$): $$ Y_{ij} - \bar{Y}_{\cdot\cdot} = (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot}) + (Y_{ij} - \bar{Y}_{i\cdot}) $$ Squaring both sides and summing over $i$ and $j$, we get \begin{align*} \sum_i \sum_j (Y_{ij} - \bar{Y}_{\cdot\cdot})² = \sum_i \sum_j (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})² + \sum_i & \sum_j (Y_{ij} - \bar{Y}_{i\cdot})² \\ & + \, 2 \sum_i \sum_j (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot}) (Y_{ij} - \bar{Y}_{i\cdot}) \end{align*}

Observe that

• $(\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})^2$ does not depend on $j$, so that $\sum_i \sum_j (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})² = \sum_i n_i (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})^2$, with $n_i$ the number of observations in group $i$.
• $ \sum_i \sum_j (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot}) (Y_{ij} - \bar{Y}_{i\cdot}) = \sum_i [ (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot}) \overbrace{\sum_j(Y_{ij} - \bar{Y}_{i\cdot})}^{=0} ] = 0 $ since the sum of deviations around the arithmetic mean is always zero.

Thus, $$ \sum_i \sum_j (Y_{ij} - \bar{Y}_{\cdot\cdot})² = \sum_i n_i (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})² +\sum_i \sum_j (Y_{ij} - \bar{Y}_{i\cdot})² $$ i.e., $$ \textrm{total sum of squares} = \textrm{between-group sum of squares} + \textrm{residual sum of squares} $$

Answer 2

If all samples come from the same population, then the means of these samples can be seen as a sampling distribution. The variance of this sampling distribution is a function of the population variance and the sample size of the drawn samples (for now we can assume the sample size is constant):

$$\sigma_\bar{x}^2 = \frac{\sigma^2}{n}$$

But what we are interested in is the population variance, so we rearrange the formula to:

$$\sigma^2 = \sigma_\bar{x}^2n$$

Of course we only have an estimate of the variance of the sampling distribution and thus also only an estimate for the population variance:

$$\hat\sigma^2 = \hat\sigma_\bar{x}^2n$$

If you partition the variance into the sum of squares and the degrees of freedom you end up with $n$ multiplied with the sum of squares:

$$\hat\sigma^2 = \frac{\sum_j^k (\bar{x}_j-\bar{\bar{x}})^2 n}{k-1}$$

Where $j=1,...j=k$ is one factor in the ANOVA and $\bar{\bar{x}}$ is the overall mean.

If sample size varies between groups, you can see the individual $n$ as a weighting factor. I am not absolutely sure, but if you weight the sum of squares with the individual $n$, you would need to divide by the sum of all individual $n$s (total sample size), but then you have the sum of these $n$s in the numerator still because this is now the correct sample size; so the formula reduces to the weighting.

Answer 3

Why do you need to multiply the between groups sum of squares by the number of observations in each group?

One often reads that the $F$-test compares the variance of the group means to the average variance of the groups. But in the actual equation, each group mean's variance is multiplied by the number of observations in that group:
$$ F(n_j-1, N-n_j)=\frac{\frac{1}{n_j -1}\sum_i n_i (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})^2}{\frac{1}{N -n_j}\sum_i\sum_j(Y_{ij} - \bar{Y}_{i\cdot})^2} $$ with $N=$ number of total observations.

As the $F$-distribution actually arises from the ratio of $\chi^2$ distributions (again divided by their respective degrees of freedom), the above equation seems even more questionable.

The qualitative answer is that, following the central limit theorem, $$ (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})^2 \approx \frac{1}{n_i} (Y_{ij}-\bar{Y}_{\cdot\cdot})^2$$ for most $j$.

That means, if we want to compare the variance of groups means to the groups' variance, we first need to take into account that the group means variance can already be expected to be smaller by a factor of $n_i$.

For this reason we need to multiply the between groups sum of squares by the number of obversations $n_i$ in each group. We, so to say, normalize the $\chi^2$ variate in the denominator to then compare it to the variate in the nominator.