What is "ANOVA"?

Question

Wikipedia says:

Analysis of variance (ANOVA) is a collection of statistical models used to analyze the differences between group means and their associated procedures (such as "variation" among and between groups). In ANOVA setting, the observed variance in a particular variable is partitioned into components attributable to different sources of variation.

In one way or two way ANOVA, (which are the types of ANOVA I have seen), the input variable $X$ is categorical, whose value represents the group of the sample.

in The Elements of Statistical Learning: Data Mining, Inference, and Prediction. by Trevor Hastie, Robert Tibshirani, and Jerome Friedman, ANOVA seems to be a way of modeling $E(Y|X)$ as sum of functions of various number of components of $X$. It doesn't relate to variance or partition of variance into some form. $X$ is not necessarily categorical either and may be continuous valued. Or am I missing something? Thanks!

Answer 1

One-way and two-way ANOVA are just two simple versions, but I doubt two experts on the topic would agree exactly what is central to ANOVA, treated in moderate or extreme generality.

For evidence, see Speed, T.P. 1987. What is an analysis of variance? Annals of Statistics 15: 885-910. Eleven discussions follow with a rejoinder by the author rounding it off.

Answer 2

ANOVA is a technique, not a model

Some sources refer to ANOVA as a "model" or a "collection of models" but in my view that is incorrect. The acronym ANOVA refers to the "analysis of variance", which is a statistical technique that can be applied to a variety of statistical models, rather than being a model itself. The essence of ANOVA lies in use of the law of iterated variance (or other variance decompositions) applied to regression models. Consider the general form of a homoskedastic regression model:

$$Y_i = u(\mathbf{X}_i, \boldsymbol{\beta}) + \varepsilon_i \quad \quad \quad \mathbb{E}(\varepsilon_i) = 0 \quad \quad \quad \mathbb{V}(\varepsilon_i) = \sigma^2.$$

Letting $v(\boldsymbol{\beta}) \equiv \mathbb{V}[u(\mathbf{X}_i, \boldsymbol{\beta})]$ and applying the law of iterated variance to this general model gives:

$$\begin{align} \mathbb{V}(Y_i) &= \mathbb{V}[\mathbb{E}(Y_i|\mathbf{X}_i)] + \mathbb{E} [\mathbb{V}(Y_i|\mathbf{X}_i) ] \\[6pt] &= \mathbb{V}[u(\mathbf{X}_i, \boldsymbol{\beta})] + \mathbb{E} [\sigma^2] \\[6pt] &= v(\boldsymbol{\beta}) + \sigma^2. \\[6pt] \end{align}$$

Now, the data generally allows you to estimate the variance of the response variable and error term, which gives the estimate:

$$\hat{v}(\boldsymbol{\beta}) = \hat{\sigma}_Y^2 - \hat{\sigma}^2.$$

Any hypothesised value for $\boldsymbol{\beta}$ gives a known value for the variance term $v(\boldsymbol{\beta})$, which means you can test the plausibility of a hypothesised values by looking at whether the implied variance is near the estimated variance. This is how you use ANOVA tests in regression models to test hypotheses on $\boldsymbol{\beta}$.