What is the relationship between ANOVA to compare means of several groups and ANOVA to compare nested models?

Question

I've so far seen ANOVA used in two ways:

First, in my introductory statistics text, ANOVA was introduced as a way to compare means of three or more groups, as an improvement over pairwise comparison, in order to determine if one of the means has a statistically significant difference.

Second, in my statistical learning text, I've seen ANOVA used to compare two (or more) nested models in order to determine if Model 1, which uses a subset of Model 2's predictors, fits the data equally well, or if the full Model 2 is superior.

Now I assume that in some way or another these two things are actually very similar because they're both using the ANOVA test, but on the surface they seem quite different to me. For one, the first use compares three or more groups, while the second method can be used to compare only two models. Would someone please mind elucidating the connection between these two uses?

Answer 1

In my understanding, the abstract intuition of ANOVA is the following: One decomposes the sources of variance of the observed variable in various directions and investigates the respective contributions. To be more precise, one decomposes the identity map into a sum of projections and investigates which projections/directions make an important contribution to explaining the variance and which do not. The theoretical basis is Cochran's theorem.

To be less abstract, I cast the second form mentioned by the OP into the framework just described. Subsequently, I interpret the first form as a special case of the second one.

Let us consider a regression model with $K$ explanatory variables (the full model) and compare it to the restricted model with $K-J$ variables. WLOG, the last $J$ variables of the full model are not included in the restricted model. The question answered by ANOVA is

"Can we explain significantly more variance in the observed variable if we include $J$ additional variables"?

This question is answered by comparing the variance contributions of the first $K-J$ variables, the next $J$ variables, and the remainder/unexplained part (the residual sum of squares). This decomposition (obtained e.g. from Cochran's theorem) is used to construct the F-test. Thus, one analyses the reduction (by including more variables) in the residual sum of squares of the restricted model (corresponding to the $H_0:$ all coefficients pertaining to the last $J$ variables are zero) by including more variables and obtains the F-statistic $$ \frac{ \frac{RSS_{restr} - RSS_{full}}{J} }{ \frac{RSS_{full}}{N-K} }$$ If the value is large enough, then the variance explained by the additional $J$ variables is significant.

Now, the first form mentioned by the OP is interpreted as a special case of the second form. Consider three different groups A, B, and C with means $\mu_A$, $\mu_B$, and $\mu_C$. The $H_0: \mu_A = \mu_B = \mu_C$ is tested by comparing the variance explained by the regression on an intercept (the restricted model) with the variance explained by the full model containing an intercept, a dummy for group A, and a dummy for group B. The resulting F-statistic $$ \frac{ \frac{RSS_{intercept} - RSS_{dummies}}{2} }{ \frac{RSS_{dummies}}{N-3} }$$ is equivalent to the ANOVA-test on Wikipedia. The denominator is equal to the variation within the groups, the numerator is equal to the variation between the groups. If the variation between the groups is larger than the variation within the groups, one rejects the hypothesis that all means are equal.

Answer 2

If you are doing one-way ANOVA to test if there is a significant difference between groups, then implicitly you are comparing two nested models (so there is only one level of nesting, but it is still nesting).

Those two models are:

• Model 0: The values $y_{ij}$ (with $i$ the sample number and $j$ the group number) are modeled by the estimated mean, $\hat{\beta}_0$ of the entire sample. $$ y_{ij} = \hat{\beta}_0 + \epsilon_i $$

• Model 1: The values are modeled by the estimated means of the groups.

  (and if we represent the the model by the between group variations, $\hat{\beta_j}$ , then model 0 is nested inside model 1)

  $$ y_i = \hat{\beta}_0 + \hat{\beta}_j + \epsilon_i $$

---

An example of comparing means and equivalence to nested models: let's take the sepal length (cm) from the iris data set (if we use all four variables we actually could be doing LDA or MANOVA as Fisher did in 1936)

The observed total and group means are:

$$\begin{array} \\ \mu_{total} &= 5.83\\ \mu_{setosa} &= 5.01\\ \mu_{versicolor} &= 5.94\\ \mu_{virginica} &= 6.59\\ \end{array}$$

Which is in model form:

$$\begin{array}\\ \text{model 1: }& y_{ij} = 5.83 + \epsilon_i\\ \text{model 2: }& y_{ij} = 5.01 + \begin{bmatrix} 0 \\ 0.93 \\ 1.58 \end{bmatrix}_j + \epsilon_i\\ \end{array}$$

The $\sum{\epsilon_i^2} = 102.1683$ in model 1 represents the total sum of squares.

The $\sum{\epsilon_i^2} = 38.9562$ in model 2 represents the within group sum of squares.

And the ANOVA table will be like (and implicitly calculate the difference which is the between group sum of squares which is the 63.212 in the table with 2 degrees of freedom):

> model1 <- lm(Sepal.Length ~ 1 + Species, data=iris)
> model0 <- lm(Sepal.Length ~ 1, data=iris)
> anova(model0, model1)
Analysis of Variance Table

Model 1: Sepal.Length ~ 1
Model 2: Sepal.Length ~ 1 + Species
  Res.Df     RSS Df Sum of Sq      F    Pr(>F)    
1    149 102.168                                  
2    147  38.956  2    63.212 119.26 < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

with $$F = \frac{\frac{RSS_{difference}}{DF_{difference}}}{\frac{RSS_{new}}{DF_{new}}} = \frac{\frac{63.212}{2}}{\frac{38.956}{147}} = 119.26$$

---

data set used in the example:

petal length (cm) for three different species of Iris flowers

Iris setosa            Iris versicolor      Iris virginica
5.1                    7.0                    6.3
4.9                    6.4                    5.8
4.7                    6.9                    7.1
4.6                    5.5                    6.3
5.0                    6.5                    6.5
5.4                    5.7                    7.6
4.6                    6.3                    4.9
5.0                    4.9                    7.3
4.4                    6.6                    6.7
4.9                    5.2                    7.2
5.4                    5.0                    6.5
4.8                    5.9                    6.4
4.8                    6.0                    6.8
4.3                    6.1                    5.7
5.8                    5.6                    5.8
5.7                    6.7                    6.4
5.4                    5.6                    6.5
5.1                    5.8                    7.7
5.7                    6.2                    7.7
5.1                    5.6                    6.0
5.4                    5.9                    6.9
5.1                    6.1                    5.6
4.6                    6.3                    7.7
5.1                    6.1                    6.3
4.8                    6.4                    6.7
5.0                    6.6                    7.2
5.0                    6.8                    6.2
5.2                    6.7                    6.1
5.2                    6.0                    6.4
4.7                    5.7                    7.2
4.8                    5.5                    7.4
5.4                    5.5                    7.9
5.2                    5.8                    6.4
5.5                    6.0                    6.3
4.9                    5.4                    6.1
5.0                    6.0                    7.7
5.5                    6.7                    6.3
4.9                    6.3                    6.4
4.4                    5.6                    6.0
5.1                    5.5                    6.9
5.0                    5.5                    6.7
4.5                    6.1                    6.9
4.4                    5.8                    5.8
5.0                    5.0                    6.8
5.1                    5.6                    6.7
4.8                    5.7                    6.7
5.1                    5.7                    6.3
4.6                    6.2                    6.5
5.3                    5.1                    6.2
5.0                    5.7                    5.9

Answer 3

Usage of ANOVA in comparison between several models means to test whether at least one of the coefficients used in model with higher order (and absent in model with the lower order) is significantly different from zero.

That is equivalent to saying that the sum of residuals for the higher order model is significantly less than that of the lower order model.

It is about two models since the basic equation used is

MSM/MSE

Where MSM is the mean of squared residuals of the lower order model (where the lowest order is the mean of target variable, i.e., intercept).

(http://www.stat.yale.edu/Courses/1997-98/101/anovareg.htm)

You can read though similar topics on CV, like

How to use anova for two models comparison?

Answer 4

From what I've learned,

You can use ANOVA tables to determine whether your explanatory variables actually have a significant affect on the response variable, and thus fit the appropriate model.

For example, suppose you have 2 explanatory variables $x_1$ and $x_2$, but you are not sure whether $x_2$ actually has an effect on Y. You can compare ANOVA tables of the two models:

$$ y=\beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon $$ vs $$ y=\beta_0 + \beta_1x_1 + \epsilon $$

You perform a hypothesis test with the Extra Residual Sum of Squares using the F-test to determine whether a reduced model with just $x_1$ is more significant.

Here is an ANOVA output example for a project I am working on in R, where I test two models (one with the Variable Days, and one without the Variable Days):

As you can see, the corresponding p-value from the F-test is 0.13, which is greater than 0.05. Thus, we cannot reject the null hypothesis that Days has no effect on Y. So, I choose model 1 over model 2.