5
$\begingroup$

According to wikipedia, in regression problems, consider 2 models, the F-statistic is $$F=\frac{(RSS_1-RSS_2)/(p_2-p_1)}{RSS_2/(n-p_2)}$$, where $p_1,p_2$ are the degrees of freedom of the 2 models respectively, $n$ is the sample size.

I can't understand $F$ intuitively, since I see that $F$ seems to be ratio between RSE of the 2 models, and it is used to choose the model which better fits the data, right? How to interpret the numerator in $F$, $(RSS_1-RSS_2)/(p_2-p_1)$?

$\endgroup$

2 Answers 2

7
$\begingroup$

Rewriting that equation makes it more intuitive:

$$F=\frac{(RSS_1-RSS_2)/RSS_1}{(df_1-df_2)/df_2}$$

The df (degrees of freedom) for each model is n-k, where n is the number of data points and k is the number of parameters fit by that model (different for the two models).

This form of the equation makes it easier to interpret. The F ratio compares the fractional decrease in sum-of-squares going from the simpler ("1") to the more complicated ("2") models as compared to the fractional decrease in degrees of freedom. When the relative decrease in sum-of-squares is much greater than the relative decrease in df, the F ratio is high, the P value is low, and you conclude that the evidence favors the more complicated model.

$\endgroup$
2
  • 1
    $\begingroup$ You write "The df (degrees of freedom) for each model is n-k, " but the equation includes df1 and df2. $\endgroup$
    – Vicki B
    Commented Oct 22, 2019 at 6:19
  • $\begingroup$ df1 = number of Independent variables (k); df2 = N - k -1, where N is the sample size, k is number of independent variables. $\endgroup$ Commented Oct 22, 2019 at 17:43
6
$\begingroup$

Let's take each portion separately.

$RSS_1 - RSS_2$ is a measure of which model has a smaller sum of squares; smaller sums of squares indicate a better model. However, sums of squares go up with sample size (other things being equal) so we need to scale it. There are different ways we could do this, but one way is by dividing by the degrees of freedom, which are closely linked to sample size, hence $p_2 - p_1$ in the denominator of the numerator.

But we don't just want to know if model 1 is better than model 2, we want to know how much better, so, again, we scale it by turning it into a proportion of the scaled RSS for model 2.

$\endgroup$
2
  • 1
    $\begingroup$ In overfitting, we also have smaller sums of squares, but it doesn't indicate a good fit, right? $\endgroup$
    – avocado
    Commented Feb 17, 2014 at 12:20
  • 1
    $\begingroup$ It does indicate a better fit, but it's a false fit - it won't generalize. $\endgroup$
    – Peter Flom
    Commented Feb 17, 2014 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.