16
$\begingroup$


I'm trying to understand the logic behind the ANOVA F-test in Simple Linear Regression Analysis. The question I have is like follows. When the F value, i.e. MSR/MSE is large we accept the model as significant. What is the logic behind this?

$\endgroup$
20
$\begingroup$

In the simplest case, when you have only one predictor (simple regression), say $X_1$, the $F$-test tells you whether including $X_1$ does explain a larger part of the variance observed in $Y$ compared to the null model (intercept only). The idea is then to test if the added explained variance (total variance, TSS, minus residual variance, RSS) is large enough to be considered as a "significant quantity". We are here comparing a model with one predictor, or explanatory variable, to a baseline which is just "noise" (nothing except the grand mean).

Likewise, you can compute an $F$ statistic in a multiple regression setting: In this case, it amounts to a test of all predictors included in the model, which under the HT framework means that we wonder whether any of them is useful in predicting the response variable. This is the reason why you may encounter situations where the $F$-test for the whole model is significant whereas some of the $t$ or $z$-tests associated to each regression coefficient are not.

The $F$ statistic looks like

$$ F = \frac{(\text{TSS}-\text{RSS})/(p-1)}{\text{RSS}/(n-p)},$$

where $p$ is the number of model parameters and $n$ the number of observations. This quantity should be referred to an $F_{p-1,n-p}$ distribution for a critical or $p$-value. It applies for the simple regression model as well, and obviously bears some analogy with the classical ANOVA framework.

Sidenote. When you have more than one predictor, then you may wonder whether considering only a subset of those predictors "reduces" the quality of model fit. This corresponds to a situation where we consider nested models. This is exactly the same situation as the above ones, where we compare a given regression model with a null model (no predictors included). In order to assess the reduction in explained variance, we can compare the residual sum of squares (RSS) from both model (that is, what is left unexplained once you account for the effect of predictors present in the model). Let $\mathcal{M}_0$ and $\mathcal{M}_1$ denote the base model (with $p$ parameters) and a model with an additional predictor ($q=p+1$ parameters), then if $\text{RSS}_{\mathcal{M}_1}-\text{RSS}_{\mathcal{M}_0}$ is small, we would consider that the smaller model performs as good as the larger one. A good statistic to use would the ratio of such SS, $(\text{RSS}_{\mathcal{M}_1}-\text{RSS}_{\mathcal{M}_0})/\text{RSS}_{\mathcal{M}_0}$, weighted by their degrees of freedom ($p-q$ for the numerator, and $n-p$ for the denominator). As already said, it can be shown that this quantity follows an $F$ (or Fisher-Snedecor) distribution with $p-q$ and $n-p$ degrees of freedom. If the observed $F$ is larger than the corresponding $F$ quantile at a given $\alpha$ (typically, $\alpha=0.05$), then we would conclude that the larger model makes a "better job". (This by no means implies that the model is correct, from a practical point of view!)

A generalization of the above idea is the likelihood ratio test.

If you are using R, you can play with the above concepts like this:

df <- transform(X <- as.data.frame(replicate(2, rnorm(100))), 
                                   y = V1+V2+rnorm(100))
## simple regression
anova(lm(y ~ V1, df))         # "ANOVA view"
summary(lm(y ~ V1, df))       # "Regression view"
## multiple regression
summary(lm0 <- lm(y ~ ., df))
lm1 <- update(lm0, . ~ . -V2) # reduced model
anova(lm1, lm0)               # test of V2
$\endgroup$
  • $\begingroup$ @chl - First of all, nice answer! This may warrant it's own question so let me know...but the descriptions I have read about ANOVA tables for regression models typically refer to three rows in the table: predictors, errors, and total. However, the anova() function in R returns an individual row for each predictor in the model. For instance, anova(lm0) above returns a row for V1, V2, and Residuals (and no total). As such, we get two F* statistics for this model. How does this change the interpretation of the F* statistic reported in the ANOVA table? $\endgroup$ – Chase Mar 14 '11 at 0:54
  • $\begingroup$ @Chase Yes, the ANOVA Table I have in mind are arranged in this way too. Feel free to ask the question; I'd love to hear what other users think of that. I generally use anova() for GLM comparison. When applied to an lm or aov object, it displays separate effects (SS) for each term in the model and doesn't show TSS. (I used to apply this the other way around, namely after fitting an ANOVA with aov(), I can use summary.lm() to get an idea of treatment contrasts.) However, there're subtle issues between summary.lm() and summary.aov(), especially related to sequential fitting. $\endgroup$ – chl Mar 14 '11 at 9:11
  • $\begingroup$ @Chase I just rediscovered this very nice response from @Gavin about the Interpretation of R's lm() output. $\endgroup$ – chl Mar 14 '11 at 11:35
  • $\begingroup$ @chl - A bit of nitpicking from me. It is a nice answer about the intuition behind the F-test, and how it "goes in the right directions". But it doesn't explain the logic of why you should choose this particular test. For example, why shouldn't we use the PRESS statistic? You hinted at the likelihood ratio - which does have a logical justification - hence its applicability to all models, unlike the F-test. $\endgroup$ – probabilityislogic Mar 15 '11 at 13:23
  • $\begingroup$ @probabilityislogic Good point. My idea was originally to show the logic behind model comparison, of which the simple regression model is just a particular case (compare to the "very null" model), which also motivates the quick note about LRT. I agree with you, if we work along the line of a pure Neyman-Pearson approach for HT. However, I was mainly thinking in terms of the Theory of LMs, where SS have a direct geometrical interpretation and where model comparison or the single F-test for a one-way ANOVA (...) $\endgroup$ – chl Mar 15 '11 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.