Mathematical Explanations behind ANOVA I was wondering: could someone could direct me to papers (preferably) or textbooks that give strong explanations of the mathematical background behind ANOVA?
 A: Try: Linear Models by Shayle Searle
A: I don't know if mathy enough (but it should have the right references to get you started):
Gelman, A. (2005). Analysis of Variance: Why It Is More Important than Ever. The Annals of Statistics, 33(1), 1–31. doi:10.2307/3448650
Should be available from Andrew Gelman's website.
A: Here is a little note I wrote for myself when learning ANOVA. Hope it helps clarify things bit.
ANOVA is the classical method to compare means of multiple ($\ge 2$) groups. Suppose $N$ observations were sampled from $k$ groups and define $n=N/k$. Let $x_{ij}$ be the $j$th observation from the $i$th group. Here we assume a balanced design i.e the number of samples from each group remain the same. Denote $\bar{x}_.$ the grand sample mean and $\bar{x}_i$ the sample mean of group $i$. Observations can be re-written as 
$$x_{ij} = \bar{x}_. + \left(\bar{x}_i - \bar{x}_.\right) + \left(x_{ij}-\bar{x}_i\right)$$
This leads to the following model
$$x_{ij} = \mu + \alpha_i + \epsilon_{ij}$$
where $\mu$ and $\alpha_i$ are grand mean and $i$th group mean respectively. The error term $\epsilon_{ij}$ is assumed to be iid from a normal distribution
$$\epsilon_{ij} \sim N(0,\sigma^2)$$
The null hypothesis in ANOVA is that all group means are the same i.e 
$$\alpha_1 = \alpha_2 = \ldots = \alpha_k$$
If this is true, the error term for group differences $\bar{x}_i - \mu \sim N(0,\sigma^2/n=\bar{\sigma}^2).$ However, you cannot directly test this by using one-sample t-test (discard $x_{ij}$ and only use $\bar{x}_i$). Suppose you have  $\sigma^2=5$ and $\bar{\sigma}^2 = 1000$ e.g between group difference is much larger than within group difference. In this case, data from individual groups are similar but groups are quit different, so we should reject the null hypothesis although the one sample t-test may fail to reject. It is really the RELATIVE magnitude of within & between group differences matters. You cannot say much by only looking at one of them.
Now consider the sum of squares for between group difference
$$\mathrm{SSD_B} = \sum_{i=1}^k \sum_{j=1}^n \left(\bar{x}_i - \bar{x}_.\right)^2 = n\sum_{i=1}^k \left(\bar{x}_i - \bar{x}_.\right)^2$$
and for within group difference
$$\mathrm{SSD_W} = \sum_{i=1}^k\sum_{j=1}^n \left(x_{ij} - \bar{x}_i \right)^2$$
where $\mathrm{SSD_B}$ has a degree of freedom of $k-1$ and $\mathrm{SSD_W}$ has a degree of freedom of $N-k$. If there is no systematic difference between the groups, we would expect the mean squares 
$$\mathrm{MS_B} = \mathrm{SSD_B}/(k-1)$$
$$\mathrm{MS_W} = \mathrm{SSD_W}/(N-k)$$
would be similar. The test statistic in ANOVA is defined as the ratio of the above two quantities:
$$F = \mathrm{MS_B}/\mathrm{MS_W}$$
which follows a F-distribution with $k-1$ and $N-k$ degrees of freedom. If null hypothesis is true, $F$ would likely be close to 1. Otherwise, the between group mean square $\mathrm{MS_B}$ is likely to be large, which results in a large $F$ value. Basically, ANOVA examines the two sources of the total variance and sees which part contributes more. This is why it is called analysis of variance although the intention is to compare group means.
For unbalanced designs and general discussion, you can look at Introducing Anova and Ancova: A GLM Approach by Andrew Rutherford. 
