I'm reading an open-source statistics book and I came across this quote:

In multiple regression, the degrees of freedom associated with the variance of the estimate of the residuals is $n - k - 1$, not $n - 1$. For instance, if we were to make predictions for new data using our current model, we would find that the unadjusted $R^2$ is an overly optimistic estimate of the reduction in variance in the response, and using the degrees of freedom in the adjusted $R^2$ formula helps correct this bias.

A couple of quick questions:

  1. Does anyone have a better and more intuitive reason behind what is degrees of freedom?
  2. Why is the reason that R^2 is an overly optimstic estimate of the reduction of variance in the response (What does this even mean?)? What is wrong with saying the degrees of freedom is $n - 1$? Why do we need to make this correction?

Here is the actual page:

enter image description here

  • $\begingroup$ "the variance of the estimate of the residuals" is the parameter or the function of the parameters. So the variance of the estimate of the residuals has no the degree of freedom. So my suggestion is ignoring this book. $\endgroup$ – user158565 Apr 29 '17 at 20:43
  • $\begingroup$ This isn't clarifying. What is "the parameter or the function of the parameters"? $\endgroup$ – Jwan622 Apr 29 '17 at 20:44
  • $\begingroup$ For example, a normal distribution has two parameters: mean and variance $\sigma^2$. Both of them are (unknown) constants. When we have a sample with n subjects from this distribution, we can use the sample mean to estimate the population mean. "the variance of the estimate" of the mean is $\sigma^2/n$. $\endgroup$ – user158565 Apr 29 '17 at 20:51
  • $\begingroup$ Reference please, not just allusion and quotation. "a less estimate"? $\endgroup$ – Nick Cox Apr 30 '17 at 13:09
  • $\begingroup$ Yeah this book seems bad. $\endgroup$ – user2879934 Apr 30 '17 at 13:12

There are lots of resources for how to think about degrees of freedom. Personally, I like thinking about it as just the number of data points - the number of parameters. When you take a sample mean $$\bar{X}=\frac{1}{n}\sum X_i$$ you are estimating a quantity without any parameters, so there are $n$ dfs.

For variance $$\frac{1}{n-1}\sum_i(X_i-\bar{X})^2$$ you are using $\bar{X}$ as a parameter that is estimated from the data so the df is $n-1$. There are lots of ways to think about it so I will move on to your next question.

Consider $$R= 1 - \frac{Var(e_i)}{Var(Y_i)}$$ But what is $Y_i$ in linear regression? Well it is just $$Y_i= \beta X_i+\epsilon$$ let's define $$\hat{Y} = \beta X_i$$ then we can write the difference between our observed $Y_i$ and our guess for $\hat{Y_i}$ as $$Y_i-\hat{Y_i}$$. But wait a second, that can be negative, and negative error doesn't make sense? Let's just square it so it will always be positive :) Okay so we get $$(Y-\hat{Y})^2$$ Now, pretend you had never heard of linear regression and someone came to you with some data on house prices and school quality and said guess the house price at this level of school quality. Well, you have no idea what to do with school quality so you ignore it and just take the average of the house prices, call that $$\bar{Y}$$ This is not a bad guess, especially since you didn't learn about regression. So how much better is our dumb guess than the complicated regression stuff? Well what if we wrote the error as $$(Y_i-\hat{Y_i})^2+(\hat{Y_i}-\bar{Y})^2$$ So if $\hat{Y_i}$ is just as good a guess as $\bar{Y}$ then the second term goes to zero, but if it is much better than the first term will go zero. Now we have a tradeoff between how using our school district quality increases the quality of our guess (given by the second term), and how much noise is left in the data that our regression line can't pick up (given by the first term).

Okay what does this have to do with $R^2$, well the actual definition above is $$R^2 = 1- \frac{\frac{1}{n}\sum_i(Y_i-\hat{Y_i})^2}{\frac{1}{n}\sum_i(\hat{Y_i}-\bar{Y})^2+\frac{1}{n}\sum_i(Y_i-\hat{Y_i})^2} $$ Why? Well the numerator is the sample variance of our errors, in other words, the variance of the error that our regression line can't capture and the denominator is the variance of the observations.

What do we see there ? $\frac{\frac{1}{n}}{\frac{1}{n}}$ That cancels, giving you the definition in the book. However those are our degrees of freedom, $n$ for the numerator and $n$ for the denominator.

Okay so what happens if we add in another explanatory variable, like number of crimes in the area? Let's look at the numerator of $R^2$. Can this amount ever increase? The objective function of regression is to find coefficients that minimize $$\sum_i(Y_i-\hat{Y_i})^2$$, so given an optimal parameter setting for $b_1$ $$\sum_i(Y_i-(b_1x_1 +b_2x_2))^2$$ the algorithm will find $b_2$ that minimizes this total sum, so there is no way this term will increase because in the worst case regression will just set $b_2=0$. Only if $x_2$ helps explain the residual error better will $b_2 \neq 0$. If we decrease the numerator, we also decrease the second term in the denominator. Lets investigate how this affects $R^2$ $$\frac{x}{x+y} \geq?\leq \frac{x-\delta}{x-\delta+y}$$ $$x(x-\delta+y) \geq?\leq (x-\delta)(x+y)$$ $$x^2-x\delta+xy \geq?\leq x^2-\delta x +xy -\delta y$$ $$0 \geq?\leq -\delta y$$ Since both $y$ and $\delta$ are $>0$ we see that the final inequality must be $0 \geq -\delta y$ so therefore, subtracting a $\delta$ from the numerator and the denominator makes the whole term go down, and therefore makes $R^2=1-that$ go up.

There is exactly the problem with $R^2$, we can artificially inflate it by just adding more parameters to the model. How do we fix this? We just add a term that penalizes models with large number of parameters.

By dividing the numerator by $n-k-1$ where $k$ is the number of parameters(regressors) in our model, we are dividing by a smaller number in the numerator than in the denominator, so the numerator becomes bigger and $R^2$ goes down! Now we have a way of making $R^2$ go down by including parameters that have no correlation.

| cite | improve this answer | |
  • $\begingroup$ Why would we write the error as: $$(Y_i-\hat{Y_i})^2+(\hat{Y_i}-\bar{Y})^2$$ $\endgroup$ – Jwan622 Apr 30 '17 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.