Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

According to wikipedia, in regression problems, consider 2 models, the F-statistic is $$F=\frac{(RSS_1-RSS_2)/(p_2-p_1)}{RSS_2/(n-p_2)}$$, where $p_1,p_2$ are the degrees of freedom of the 2 models respectively, $n$ is the sample size.

I can't understand $F$ intuitively, since I see that $F$ seems to be ratio between RSE of the 2 models, and it is used to choose the model which better fits the data, right? How to interpret the numerator in $F$, $(RSS_1-RSS_2)/(p_2-p_1)$?

share|improve this question

2 Answers 2

Let's take each portion separately.

$RSS_1 - RSS_2$ is a measure of which model has a smaller sum of squares; smaller sums of squares indicate a better model. However, sums of squares go up with sample size (other things being equal) so we need to scale it. There are different ways we could do this, but one way is by dividing by the degrees of freedom, which are closely linked to sample size, hence $p_2 - p_1$ in the denominator of the numerator.

But we don't just want to know if model 1 is better than model 2, we want to know how much better, so, again, we scale it by turning it into a proportion of the scaled RSS for model 2.

share|improve this answer
    
In overfitting, we also have smaller sums of squares, but it doesn't indicate a good fit, right? –  loganecolss Feb 17 at 12:20
    
It does indicate a better fit, but it's a false fit - it won't generalize. –  Peter Flom Feb 17 at 12:48

Rewriting that equation makes it more intuitive:

$$F=\frac{(RSS_1-RSS_2)/RSS_1}{(df_1-df_2)/df_2}$$

The df (degrees of freedom) for each model is n-k, where n is the number of data points and k is the number of parameters fit.

This form of the equation makes it easier to interpret. The F ratio compares the fractional decrease in sum-of-squares going from the simpler ("1") to the more complicated ("2") models as compared to the fractional decrease in degrees of freedom. When the relative decrease in sum-of-squares is much greater than the relative decrease in df, the F ratio is high, the P value is low, and you conclude that the evidence favors the more complicated model.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.