Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm trying to understand the exact meaning of the $F$-values and what are we testing in the ANOVA table for a simple linear model in R:

> asdf=lm(carb~weight+protein+age)
> anova(asdf)
Analysis of Variance Table

Response: carb
          Df Sum Sq Mean Sq F value  Pr(>F)   
weight     1 181.38 181.378  5.1123 0.03804 * 
protein    1 305.40 305.400  8.6079 0.00973 **
age        1  38.36  38.359  1.0812 0.31389   
Residuals 16 567.66  35.479                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

If we have two nested linear models $A$ in $B$ with $p_0$ and $p$ covariates respectively, we use the $F$-statistic obtained from the likelihood ratio test: $$ \frac{(n-p-1)}{(p-p_0)}\cdot\frac{RSS(A)-RSS(B)}{RSS(B)} $$ to find if we prefer $B$ over $A$. The $F$-value 1.0812 in the third line makes sense, as it is exactly this statistic to compare the full model and the model omitting the covariate age. We could justify omitting the age covariate by this result.

I'm puzzled about $F$ values in the first two lines. I see the first value is obtained by taking ratio of: $$ 16\cdot\frac{(RSS({\rm null})-RSS({\rm incl.weight})}{RSS({\rm full model})} $$ and the second line is similar. What exact hypothesis is this testing, and how is it more logical than using: $$ 18\cdot\frac{(RSS({\rm null})-RSS({\rm incl.weight})}{RSS({\rm incl.weight})} $$ as for the third value in the explanation above?

I think my question wasn't completely clear, so I'll clarify. Let's say we try further:

> a=lm(carb~1)
> as=lm(carb~weight)
> anova(a,as)
Analysis of Variance Table

Model 1: carb ~ 1
Model 2: carb ~ weight
  Res.Df     RSS Df Sum of Sq      F Pr(>F)  
1     19 1092.80                             
2     18  911.42  1    181.38 3.5821 0.0746 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Why when we use anova on the full model alone, we get $F$ value for weight vs null $5.1123$, different from $3.5821$ here, and how are these different?

share|improve this question
add comment

2 Answers 2

up vote 3 down vote accepted

For age, $F=1.0812=\frac{MS_\text{age}}{MS_\text{Residuals}}$. The same is true of the other $F$ values replacing age with weight or protein. This partials out the variance in carb that is related to the other two factors not being tested directly by the $F$ test in question, whereas using $MS_\text{Residuals}$ from a single-factor GLM in the denominator would not. Thus the hypothesis test is of whether the residual variance in carb that is not explained by your other factors can be explained by your given factor. More specifically, the $p$ value represents the probability that this residual variance would relate to your given factor at least as much as it does in your sample if you were to sample again randomly from a population in which the null hypothesis of no relationship between those residuals and your factor is literally true. As for why 16 and not 18, remember that controlling for these other factors costs you degrees of freedom: one apiece.

To elaborate in response to your edit/comment, another way of looking at your $F=3.5821$ is as $F=\frac{MS_\text{weight}}{MS_\text{Residuals}}$ for a general linear model with only one factor (weight). With that one-factor GLM's ${MS_\text{Residuals}}$ instead of a three-factor GLM's ${MS_\text{Residuals}}$ as the denominator (because you're not controlling for anything, hence what would've been the residuals are the observations $-\ \mu_\text{carb}$ instead, $\mu_\text{carb}$ being the intercept of the null model), you haven't partialed out any of the variance that weight can't explain but age or protein can, so the effect of weight appears less clear by itself.

When you control for the effects of age and protein, you reduce the amount of variance that still needs explaining in your model. This makes the predictive job a little easier for weight, because it no longer has to contend with the independent effects of age or protein in explaining carb. In a post hoc / retrospective sense, you can look back and say, "Well, no wonder weight didn't predict these observations as well by itself; age and protein vary in my sample too, and their independent effects were mucking things up for poor ol' weight!" Of course, these results are even better in an epistemic, hypothesis-testing sense if you expected in advance that this would happen, and chose multiple regression to examine hypotheses of independent effects you also specified in advance.

share|improve this answer
    
I think my initial question wasn't clear enough, so I edited it. The main point is, why are these F statistics different from ones we would obtain by using anova(lm(null),lm(null+weight)) for example? And are the hypothesis different for these? –  pk33333 Mar 21 at 19:44
    
Noted and elaborated to respond. –  Nick Stauner Mar 21 at 20:03
    
I don't completely follow your answer, but I'm not sure it's right. The F for the full model is not MS(full)/MS(total), it is [{RSS(reduced)-RSS(full)}/{Rdf(reduced)-Rdf(full)}] / RMS(full). This is true whether the reduced model is the null model or not. I'm really having trouble seeing how MS(total) comes in anywhere. –  gung Mar 21 at 22:42
    
Ugh. Sorry, I thought I had this one sorted out. I kind of avoided trying to interpret the OP's "full model", because I would've thought this to be the model based on all three predictors, and I don't see that where "full model" is mentioned, so I'm not sure what it means. On second glance though, yeah, I meant $MS_\text{Error}$ in the ANOVA sense (maybe still "residuals"). Is it ironic that this was my error? I don't even know anymore. :P Let me know if you catch anything else, and sorry for the trouble! I've always relied somewhat heavily on my eraser in matters of mathematical reasoning... –  Nick Stauner Mar 21 at 23:13
1  
Calling whichever is the larger model the "full model", & whichever is the smaller model the "reduced model", is just standard generic terminology. In this case, after the edit, as is the full model & a is the reduced model. –  gung Mar 21 at 23:17
add comment

The anova function in R, when given only 1 model, produces the "Sequential" sums of squares. This means that each term is adjusted for those above it (to the left in the model formula) and don't include those below (to the right). So in your example you are testing weight vs. intercept only, protein + weight vs. weight only, and age + protein + weight vs. protein + weight.

If you change the order of the terms in the formula statement then it will change the conditioning in those tests.

If you give anova 2 models (with one of them nested in the other) then it will give the full and reduced test with the null that the smaller model is sufficient and the alternative that the additional terms in the fuller model do contribute.

share|improve this answer
    
I think my question wasn't completely clear, so I edited it. –  pk33333 Mar 21 at 19:33
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.