5
$\begingroup$

After reading lots of articles and tutorials on the Internet, I can probably get the idea of the degree of freedom of some kind of models (e.g., regression model.)

However, I have another question. After knowing the df of a model, how should we use it? And why is this number useful?

When using the Python package statsmodels to summarize a data set, this tool outputs both the df of residuals and the df of the model. Why should we need this information while assessing the model?

$\endgroup$
4
  • $\begingroup$ possible duplicate of How to understand degrees of freedom? $\endgroup$
    – Tim
    Sep 10, 2015 at 9:13
  • 1
    $\begingroup$ @Tim: Useful link, but I think the focus on regression models distinguishes this question. $\endgroup$ Sep 10, 2015 at 9:27
  • $\begingroup$ Yes, I dont think this is a duplicate. vote to stay open. $\endgroup$ Sep 10, 2015 at 9:38
  • $\begingroup$ Would you consider accepting my answer or commenting if you need further clarification. $\endgroup$
    – Gumeo
    Sep 23, 2015 at 11:37

1 Answer 1

6
$\begingroup$

When dealing with predictive models it is maybe better in some sense to think about the number of parameters in the model. The number of parameters shows in some sense how flexible the model is. Parameters may be dependent, e.g. in hierarchical models, so then you need to look at the effective number of parameters, which is another way to quantify the flexibility of the model.

This is mostly to account for overfitting, (although that is not the whole truth).

Imagine that you are fitting an n-th degree polynomial to n+1 data points. The polynomial has n+1 parameters and will hit every single one of your data points. The polynomial may have huge parameters and fluctuate very high up and down. This is probably not the true underlying model in most cases.

Thus you can for example regularize the parameters, e.g. by penalizing the norm of the parameters. This reduces the effective number of parameters, thus restricting the degrees of freedom in the model. Another option is to fit a lower degree polynomial and see how that looks.

If a model has a degree of freedom $p$, you would need at least $p$ data points to get an estimate of the parameters in the model, otherwise you have an underdetermined system. If you are fitting some $n$ data points with large errors you usually want $n$ to be pretty much larger than $p$. Otherwise you risk overfitting. The case where it is "ok" to have $n$ close to $p$ is when the errors are very small and you really know the true underlying model, which in most cases is not true.

Degrees of freedom for test statistics is the number $\nu=n-p$, so they are not entirely the same thing but very closely related.

To summarize

So if the degrees of freedom in your model are on the scale of the number of data points, you will most likely overfit and have very bad predictions.

This blog summarizes this quite well.

To completely understand degrees of freedom, in the sense of tests and parameter estimates, check out this CV post

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.