2
$\begingroup$

I'm reading on a text book about linear regression, and when I thought I finally understood degrees of freedom, I found a statement that made me doubt what I know so far. Well it's in the context of a simple linear regression (1 input).

The orange, blue and green squares indicate the MSEs associated with the corresponding curves in the lefthand panel. A more restricted and hence smoother curve has fewer degrees of freedom than a wiggly curve—note that in Figure 2.9, linear regression is at the most restrictive end, with two degrees of freedom.

What degree of freedom they talking about in here? What I know is that: df(regression)=p with p is the number of features used. So for this case it must be 1. df(residuals)=n-(p+1). df(total)=n-1. n is the sample size. Any help on what that 2 might be? Here is the two panels of the figure 2.9. Right hand panel Left hand panel

$\endgroup$
  • $\begingroup$ Could you give us the full quote and name of the textbook? $\endgroup$ – Tim Oct 30 '18 at 10:58
  • $\begingroup$ Yes of course, it's from Introduction to statistical learning. The full quote is: "A more restricted and hence smoother curve has fewer degrees of freedom than a wiggly curve—note that in Figure 2.9, linear regression is at the most restrictive end, with two degrees of freedom." $\endgroup$ – Youssef Esseddiq Oct 30 '18 at 11:01
  • 1
    $\begingroup$ My guess is the authors counted 1 for the intercept and 1 for the slope (aka, the 1 you took from n in your question). The orange line is straight so it should be a one-predictor regression. $\endgroup$ – Penguin_Knight Oct 30 '18 at 11:59
  • $\begingroup$ I am more impressed that you understand the degrees of freedom of the locally fitting model, @YoussefEsseddiq :). Way harder to compute, too. $\endgroup$ – StasK Oct 30 '18 at 13:41
  • $\begingroup$ In the first graph, x axis is labeled "Flexibility". What does it mean? Another name for degree of freedom? $\endgroup$ – user158565 Oct 30 '18 at 14:09
3
$\begingroup$

I second @Penguin_Knight 's assertion in the comments: The second loss of d.f. comes from the intercept term in $\hat{y}=\hat{\beta_0} + \hat{\beta_1}x_1$. Every term you must estimate removes a degree of freedom from the residuals.

Most texts I have seen denote $\text{df(residuals)} = n - p$, where $p$ is the number of parameters you estimated in the model (so for linear regression, the number of covariates plus one).

$\endgroup$
1
$\begingroup$

This is about overfitting. You can fit a higher order polynominal to the curve and archive lower MSE (1st Figure). In your case you have a simple function $f(x)$ with $x$ having 1 dimension. Now you can try to fit a linear curve $\hat f(x) = ax + b$ or a higher order polynomal like $\hat f(x) = ax^3 + bx^2 + cx + d$ using linear regression (2nd Figure). You are still learning linear parameters $a,b,c,d$ by solving $\min_{a,b,c,d}(\hat f(x) - y)^2$, hence linear regression.

However, this might overfit on the training data and perform worse on your actual use case. (1st Figure)

$\endgroup$
  • 1
    $\begingroup$ I think OP is asking why the simplest model appears to be a straight line but in the text it was described to have 2 degrees of freedom. $\endgroup$ – Penguin_Knight Oct 30 '18 at 12:08
  • $\begingroup$ I quess you are right. Your comment is already the answer then. $\endgroup$ – Thomas Pinetz Oct 30 '18 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.