3
$\begingroup$

According to wikipedia, in regression problems, consider 2 models, the F-statistic is $$F=\frac{(RSS_1-RSS_2)/(p_2-p_1)}{RSS_2/(n-p_2)}$$, where $p_1,p_2$ are the degrees of freedom of the 2 models respectively, $n$ is the sample size.

I can't understand $F$ intuitively, since I see that $F$ seems to be ratio between RSE of the 2 models, and it is used to choose the model which better fits the data, right? How to interpret the numerator in $F$, $(RSS_1-RSS_2)/(p_2-p_1)$?

$\endgroup$
7
$\begingroup$

Rewriting that equation makes it more intuitive:

$$F=\frac{(RSS_1-RSS_2)/RSS_1}{(df_1-df_2)/df_2}$$

The df (degrees of freedom) for each model is n-k, where n is the number of data points and k is the number of parameters fit by that model (different for the two models).

This form of the equation makes it easier to interpret. The F ratio compares the fractional decrease in sum-of-squares going from the simpler ("1") to the more complicated ("2") models as compared to the fractional decrease in degrees of freedom. When the relative decrease in sum-of-squares is much greater than the relative decrease in df, the F ratio is high, the P value is low, and you conclude that the evidence favors the more complicated model.

$\endgroup$
2
  • 1
    $\begingroup$ You write "The df (degrees of freedom) for each model is n-k, " but the equation includes df1 and df2. $\endgroup$
    – Vicki B
    Oct 22 '19 at 6:19
  • $\begingroup$ df1 = number of Independent variables (k); df2 = N - k -1, where N is the sample size, k is number of independent variables. $\endgroup$ Oct 22 '19 at 17:43
6
$\begingroup$

Let's take each portion separately.

$RSS_1 - RSS_2$ is a measure of which model has a smaller sum of squares; smaller sums of squares indicate a better model. However, sums of squares go up with sample size (other things being equal) so we need to scale it. There are different ways we could do this, but one way is by dividing by the degrees of freedom, which are closely linked to sample size, hence $p_2 - p_1$ in the denominator of the numerator.

But we don't just want to know if model 1 is better than model 2, we want to know how much better, so, again, we scale it by turning it into a proportion of the scaled RSS for model 2.

$\endgroup$
2
  • 1
    $\begingroup$ In overfitting, we also have smaller sums of squares, but it doesn't indicate a good fit, right? $\endgroup$
    – avocado
    Feb 17 '14 at 12:20
  • 1
    $\begingroup$ It does indicate a better fit, but it's a false fit - it won't generalize. $\endgroup$
    – Peter Flom
    Feb 17 '14 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.