Null hypothesis for ANOVA for regression

Question

Context: I know the "classic" ANOVA framework: we have $n$ groups and $k$ measurements of a variable $X$ for each group. Let $µ_1$, $µ_2$, ..., $µ_n$ be the mean of $X$ in each group.

Hypothesis $H_0$: $µ_1 = µ_2 = ... = µ_n$
Hypothesis $H_1$: $\exists i,j, \rm{\ such\ that \ }µ_i \neq µ_j$

Based on the effective samples, we can compute the value of "$F$", which is finally bigger or smaller than a certain threshold (calculated with $k$, $n$, and for example $\alpha=5\%$). This allows us to reject or not reject $H_0$. This is the classic ANOVA.

---

I often read about "ANOVA for regression", for testing the linearity / the validity of a linear model, but the hypothesis $H_0$ is rarely clearly described.

Question: how can we use the framework described in the first paragraph (which is about testing whether $µ_1 = µ_2 = ... = µ_n$ or not) to test the relevance of a linear model? What is the $H_0$ hypothesis in this case?

Concrete example: let's say we want to show linearity between X and Y, here is some input data:

For $x_0 = 2$, we have 4 measurements $y_{0,0} = 3.12$, $y_{0,1} = 3.27$, $y_{0,2} = 3.45$, $y_{0,3} = 3.28$.

For $x_1 = 3$, we have 4 measurements $y_{1,0} = 4.15$, $y_{1,1} = 4.37$, $y_{1,2} = 4.01$, $y_{1,3} = 3.98$.

For $x_2 = 4$, we have 4 measurements $y_{2,0} = 5.35$, $y_{2,1} = 5.58$, $y_{2,2} = 5.11$, $y_{2,3} = 5.67$.

We can do a linear regression $y = a + bx$, find $a$, $b$, $R^2$, etc.

But how to use "ANOVA for regression" in this example?

Here an hypothesis like $H_0$: $µ_1 = µ_2 = µ_3$ does not make any sense.

Answer 1

@Basj 's answer (to their own question) puts it into the mathematical context of ANOVA. Here is a much simpler version:

For simple linear regression, the null hypothesis for the ANOVA is that the regression model (fit line) is identical to a simpler model (horizontal line). In other words, the null hypothesis is that the slope is actually zero.

Also note that the term "linearity" is not really defined in the question or the answers, and can be misleading. There can be a very strong relationship between X and Y that is not linear.

Answer 2

The typical place where students first hear the term "ANOVA: Analysis of Variance" is when the class extends the two-sample t-test to many samples (three, four, a zillion) and learn to test this with an F-test instead of a t-test.

What's happening under the hood is that this classical ANOVA is a regression model. Fix one of the groups as the baseline category which will be subsumed by the intercept (your choice won't affect the p-value or F-stat). Next, test if any of the other groups have a mean different from this baseline group. This is tested by checking if any of the coefficients on the indicator variables for the non-baseline groups are significantly different from zero, using an F-test of nested regression models: an intercept-only model that assumes all means to be equal is nested within a model that has an additional coefficient parameter for each non-baseline group that allows any of those groups to have a different mean.

For three groups, the regression model would be written as:

$$ \mathbb E\left[ y_i \right] = \beta_0 + \beta_1x_{i, \text{group1}} + \beta_2x_{i, \text{group2}} $$

For $k$ groups, the regression model is:

$$ \mathbb E\left[ y_i \right] = \beta_0 + \beta_1x_{i, \text{group1}} + \beta_2x_{i, \text{group2}} +\dots + \beta_{k - 1}x_{k-1, \text{group }k-1} $$

Where this gets extended to the full linear model (and beyond) is that none of this made any real assumptions about what the $x$ features are. Therefore, if you have three variables and want to check if two of them have nonzero coefficients, you might run regressions like the following.

$$ \mathbb E\left[ y_i \right] = \beta_0 + \beta_1x_{i, 1}\\ \text{nested within}\\ \mathbb E\left[ y_i \right] = \beta_0 + \beta_1x_{i, 1} + \beta_2x_{i, 2} + \beta_3x_{i, 3} $$

Running an F-test of these nested models (exactly what we do in the basic ANOVA to test the equality of $k$ means) means that we are testing the following null and alternative hypotheses.

$$ H_0\text{: } \beta_2 = \beta_3 = 0\\ H_a\text{: } H_0 \text{ is false (at least one of }\beta_2\text{ and }\beta_3\text{ are nonzero)} $$

When you dig into the math, you'll see that this relates to comparing the residual variance from the simple model to the residual variance from the complex model in which the simple model is nested, hence the term analysis of variance.

In your specific example, you have a simple linear regression. If you want to test if the slope is nonzero, your F-test would compare the simple linear regression with both a slope and an intercept to a model with just an intercept. When you do an F-test on just one parameter, that is equivalent to a t-test on that one parameter.

The F-test winds up being much more general than just comparing the means of $k$ groups!

RELATED CROSS VALIDATED LINKS

Why is ANOVA equivalent to linear regression?

What's the difference between regression and analysis of variance?

What are chunk tests?

I personally learned this material from a course that used Alan Agresti's Foundations of Linear and Generalized Linear Models. Section $3.2.1$ develops basic ANOVA as a regression model, and then $3.3.2$ develops the general "chunk" test. (For years, I have kept that book near where I work, either at my desk at the office or near my workspace at home.)

Answer 3

One Anova test for regression that has not been mentioned yet is the full-reduced model test. This can be interpreted as a test of linearity where you are comparing a straight line relationship to a non-linear relationship.

The full-reduced model test is very general for testing any case where we have a more complex model (the full model) and a reduced model (a simpler case) and the reduced model is "nested" in the full model, i.e. all the terms in the reduced model are also in the full model. Though the idea of nesting can be flexible in that the terms do not need to be in the exact same format (e.g. an x variable that is centered and scaled in one model, but not the other still counts).

For testing linearity the full model would have multiple terms to fit a non-linear curve between $x$ and $y$. This could be a polynomial expansion on $x$ or just the traditional analysis of variance method of fitting a mean for each unique value of $x$. The reduced model then has the linear relationship (regression) between $x$ and $y$.

The null hypothesis is that the 2 models fit equally well (the full model does not fit any better than the reduced model) and the alternative hypothesis is that the additional terms in the full model to have a significant contribution. In some model formulations the null hypothesis can be expressed in terms of coefficients equaling 0, e.g. all the non-linear polynomial coefficients if using a polynomial expansion.

Answer 4

Based on the other answers, and external readings, such as How to Interpret the F-test of Overall Significance in Regression Analysis, and A Simple Guide to Understanding the F-Test of Overall Significance in Regression , The Analysis of Variance (ANOVA) table and the F-test, here is how I now understand this example:

Let's say we want to show linearity between X and Y, here is some input data:
For $x_0 = 2$, we have 4 measurements $y_{0,0} = 3.12$, $y_{0,1} = 3.27$, $y_{0,2} = 3.45$, $y_{0,3} = 3.28$.
For $x_1 = 3$, we have 4 measurements $y_{1,0} = 4.15$, $y_{1,1} = 4.37$, $y_{1,2} = 4.01$, $y_{1,3} = 3.98$.
For $x_2 = 4$, we have 4 measurements $y_{2,0} = 5.35$, $y_{2,1} = 5.58$, $y_{2,2} = 5.11$, $y_{2,3} = 5.67$.

First, the hypotheses:

Null hypothesis (H0) : The model with no predictor variables (also known as an intercept-only model) fits the data as well as the regression model.

Alternative hypothesis (H1) : The regression model fits the data better than the intercept-only model.

Least squares give: y = 1.0571 + 1.0738 x

                      df     SS           MS                   F
Regression            1      SSR          MSR = SSR / 1        F = MSR / MSE = 194.2
Residual              10     SSE          MSE = SSE / (n-2)
Total                 11     SST 

In the table https://www.stat.purdue.edu/~lfindsen/stat503/F_alpha_05.pdf, we can see that F > 4.965, thus we reject $H_0$.