# Degrees of freedom for linear regression

I'm reading on a text book about linear regression, and when I thought I finally understood degrees of freedom, I found a statement that made me doubt what I know so far. Well it's in the context of a simple linear regression (1 input).

The orange, blue and green squares indicate the MSEs associated with the corresponding curves in the lefthand panel. A more restricted and hence smoother curve has fewer degrees of freedom than a wiggly curve—note that in Figure 2.9, linear regression is at the most restrictive end, with two degrees of freedom.

What degree of freedom they talking about in here? What I know is that: df(regression)=p with p is the number of features used. So for this case it must be 1. df(residuals)=n-(p+1). df(total)=n-1. n is the sample size. Any help on what that 2 might be? Here is the two panels of the figure 2.9.

• Could you give us the full quote and name of the textbook? – Tim Oct 30 '18 at 10:58
• Yes of course, it's from Introduction to statistical learning. The full quote is: "A more restricted and hence smoother curve has fewer degrees of freedom than a wiggly curve—note that in Figure 2.9, linear regression is at the most restrictive end, with two degrees of freedom." – Youssef Esseddiq Oct 30 '18 at 11:01
• My guess is the authors counted 1 for the intercept and 1 for the slope (aka, the 1 you took from n in your question). The orange line is straight so it should be a one-predictor regression. – Penguin_Knight Oct 30 '18 at 11:59
• I am more impressed that you understand the degrees of freedom of the locally fitting model, @YoussefEsseddiq :). Way harder to compute, too. – StasK Oct 30 '18 at 13:41
• In the first graph, x axis is labeled "Flexibility". What does it mean? Another name for degree of freedom? – user158565 Oct 30 '18 at 14:09

I second @Penguin_Knight 's assertion in the comments: The second loss of d.f. comes from the intercept term in $$\hat{y}=\hat{\beta_0} + \hat{\beta_1}x_1$$. Every term you must estimate removes a degree of freedom from the residuals.
Most texts I have seen denote $$\text{df(residuals)} = n - p$$, where $$p$$ is the number of parameters you estimated in the model (so for linear regression, the number of covariates plus one).
This is about overfitting. You can fit a higher order polynominal to the curve and archive lower MSE (1st Figure). In your case you have a simple function $$f(x)$$ with $$x$$ having 1 dimension. Now you can try to fit a linear curve $$\hat f(x) = ax + b$$ or a higher order polynomal like $$\hat f(x) = ax^3 + bx^2 + cx + d$$ using linear regression (2nd Figure). You are still learning linear parameters $$a,b,c,d$$ by solving $$\min_{a,b,c,d}(\hat f(x) - y)^2$$, hence linear regression.