5
$\begingroup$

In the typical environment for multiple linear regression, we have that $Y = \mathbf{X}\beta + \epsilon$ where $\epsilon$ is iid $\mathcal{N}(0, \sigma^2I) $ where $\sigma^2$ is unknown. In this case, regression sum of squares (SSR) has $\text{df}= p - 1$ ($\text{df}$ = degrees of freedom) where $p$ is the number of parameters in the model. I have two questions based on this.

  1. Why is the $\text{df}= p - 1$, it seems to me that all $p$ of the parameters are free to vary.
  2. If $\sigma^2$ is known, is the $\text{df}= p - 2$?

Edit: I am talking about sum of squares due to regression which is defined as $\sum (\hat{y}_i-\bar{y})^2$ this has degrees of freedom p-1 where p is the number of parameters in the model. SST which is defined as $\sum (y_i-\bar{y})^2$ has degrees of freedom n-1 and SSE (sum of squares due to error/residuals) is defined as $\sum (\hat{y}_i-y_i)^2$ and has degrees of freedom n-p.

SST = SSE + SSR and $df_{SST} = df_{SSR}+df_{SSE}$.

$\endgroup$
  • 1
    $\begingroup$ Please spell out your abbreviations. Presumably, "SSR" is the sum of squares of residuals and "df" is degrees of freedom. If that's so, then how do you determine that df = p-1? Usually it has $n-p-1$ degrees of freedom when a constant is included in the regression and there are $n$ data values. That leaves me wondering whether I have correctly understood your abbreviations. $\endgroup$ – whuber Jul 3 '13 at 16:09
  • $\begingroup$ Oops! I'm so used to using SSR for Regression Sum of Squares and SSE for Error Sum of Squares that I forgot that these are not standard abbreviations. $\endgroup$ – TrynnaDoStat Jul 3 '13 at 16:16
  • $\begingroup$ About the intuition behind question #2: I am wondering why knowing $\sigma$ ought to change the df at all. Suppose you knew all the parameters of the problem: $\beta$ and $\sigma$. Obviously that does not change SSE, SSR, or SST (unless you used them to compute $\hat{y}_i$ and $\bar{y}$, but that would change their definitions). So how could your knowledge of the true parameters change the sampling distribution of the statistics (could this be a new form of ESP? :-)? $\endgroup$ – whuber Jul 3 '13 at 19:22
6
$\begingroup$

There is a common confusion in your understanding of the degrees of freedom in the typical situation. The degrees of freedom for a multiple regression is often written $\text{df}=N-p-1$. I think this is where you are getting the idea that the df for the regression is $p-1$. I typically prefer to write this as $\text{df}=N-(p+1)$ for this reason. Note that $-1*(p+1)=(-p-1)$, hence: $N-(p+1)=N-p-1$. This is a small piece of algebra that I think is slightly less intuitive for people.

Let's work though the degrees of freedom and see if we can make more sense of it:

  • If you simply have $N$ data, and are doing nothing with them, then I suppose you could say you have $N~\text{df}$.
  • If you wanted to calculate the mean of your data, then you have used $1~\text{df}$, so now you have only $N-1~\text{df}$ left. The standard way of explaining this is that all but one of your data could be any old real number, but in order to end up with the mean you've found, the last datum will be forced to be some particular value--it wouldn't be 'free to vary'.
  • So, what if you wanted to fit a simple (i.e., 1 $X$ variable) regression model? Then you are estimating the mean of $Y$, i.e. $\bar y$, and the slope of the relationship between $X$ and $Y$, $\hat\beta_1$. (Most people typically think of this as estimating the slope, $\hat\beta_1$, and the intercept, $\hat\beta_0$, but note that once you have $(\bar x,~\bar y)$ and $\hat\beta_1$, the intercept falls out of this: $\hat\beta_0=\hat\beta_1\bar x-\bar y$; and remember that $\bar x$ is free / doesn't cost any degrees of freedom, because your $X$ variable is assumed to be a set of known constants.)
  • And what if you wanted to fit a multiple regression model with $p$ $X$ variables? Then you would be estimating the mean of $Y$ and $p$ slopes, i.e., $\bar y$ and $\hat\beta_j$ (or $p$ slopes, $\hat\beta_j$, and the intercept, $\hat\beta_0$, if that's more intuitive for you). Thus, you would have $N-(p+1)~\text{df}$ remaining.
  • Importantly, recognize that nowhere in this discussion have we discussed estimating $\hat\sigma^2_\varepsilon$ yet, nor is that part of calculating the $\text{df}$ for our model.

The residual variance has to do with how we form intervals. For example, if you wanted to form a confidence interval around a slope, $\hat\beta_j$, to test whether it was equal to some null value (typically $0$). If you know the residual variance a-priori, and your residuals are normal (or, your sample is sufficiently large for the Central Limit Theorem to cover for you), then you can use the normal distribution to form this CI / to test your slope. On the other hand, if you estimated your residual variance from your data, then you would use the $t$ distribution for this, with the model's $\text{df}$.


(Response to edit:)

I don't quite follow the thinking behind your question; I continue to suspect there is a misunderstanding at play. The ANOVA table for a regression model, e.g., is constructed like this:

\begin{array}{lllll} &\text{Source} &\text{SS} &\text{df} &\text{MS} &\text{F} \\ \hline &\text{Regression} &\sum(\hat y-\bar y)^2 &p &\frac{\text{SS}_{reg}}{\text{df}_{reg}} &\frac{\text{MS}_{reg}}{\text{MS}_{res}} \\ &\text{Residual} &\sum(y_i-\hat y)^2 &N-(p+1) &\frac{\text{SS}_{res}}{\text{df}_{res}} \\ &\text{Total} &\sum(y_i-\bar y)^2 &N-1 \end{array}

To be explicit:

  • The sum of squares due to regression has $p\text{ df}$, not $p-1$.
  • Knowledge of $\sigma^2_\varepsilon$ (or the lack thereof) is unrelated to the $\text{df}$.
  • The parameters aren't free to vary, the data are.
$\endgroup$
  • 2
    $\begingroup$ +1. Very nice explanation. Feel free to dismiss this as ultra-picky, but I prefer df to $df$ as notation. Composite symbols are generally better as roman (think exp not $exp$, log, sin, cos, etc. Otherwise somebody wastes time asking what $d$ and $f$ are which are multiplied together. $\endgroup$ – Nick Cox Jul 3 '13 at 16:49
  • $\begingroup$ Looks better; thanks for listening. In many ways, "degrees of freedom" is an awful term, as it presupposes familiarity with the idea in mechanics. Also, one should often be saying "the number of degrees of freedom", which is awkward at best. $\endgroup$ – Nick Cox Jul 3 '13 at 16:58
  • $\begingroup$ @NickCox, honestly, when I say it, I just say "degrees freedom" (I don't even use the of); I tend to move randomly between sloppy & excessively meticulous ;-). $\endgroup$ – gung Jul 3 '13 at 17:01
  • $\begingroup$ I think I understand why we are disagreeing. My book is defining p as the number of parameters in the regression model INCLUDING the intercept term. The way I'm defining it, p = 2 in simple linear regression. However, I believe you are defining it as the number of non-intercept parameters in the model. So for you, p = 1 for simple linear regression. Bah, my terrible book has made this much more difficult than it needs to be. I appreciate it! $\endgroup$ – TrynnaDoStat Jul 3 '13 at 20:52
  • $\begingroup$ If so, that would be the source of the miscommunication. It is standard to refer to $p$ predictors (& thus, $p+1~\beta$s); I've never seen it presented the other way. At any rate, you're welcome. $\endgroup$ – gung Jul 3 '13 at 21:04
0
$\begingroup$

I presume you mean by your $p$ the number of data points, which is usually denoted by $n$ (parameters refers to something different in statistics and are here conventionally denoted by $p$). The data fix the relationship between the $n$ points and hence they fix the SSR as a single value (boundary condition). Therefore, you have $n-1$ degrees of freedom. If you then need to estimate $\sigma^2$ from these as well, you lose another DoF, i.e., $n-1-1$.

$\endgroup$
  • $\begingroup$ written in parallel with gung's reply. $\endgroup$ – Lucozade Jul 3 '13 at 16:46
  • 1
    $\begingroup$ Please see my edit. $\endgroup$ – TrynnaDoStat Jul 3 '13 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.