4
$\begingroup$

I read that in a regression with $k$ regressors, the t-statistic corresponding to a certain coefficient follows a $t(n-k)$ distribution. However, later on I read that studentized residuals follow a $t(n-k-1)$ distribution. How could it be that the extra degree of freedom is lost: isn't this also just based on a regular t-statistic?

$\endgroup$
  • 2
    $\begingroup$ Do you mean externally studentized residuals? These are calculated by deleting one observation, which would explain a loss of 1 degree of freedom. $\endgroup$ – ved Jul 1 '14 at 18:03
  • $\begingroup$ @ved Thanks for your reply. To be honest I'm not entirely sure what you mean with "externally" in this context. The studentized residual that I mean is the t-statistic for $\hat{\gamma}$ that can be obtained from the regression $y=X\beta + \gamma D_j$. Do you mean that the fact that there is a dummy in the regression somehow implies that the degree of freedom is lost? $\endgroup$ – rbm Jul 1 '14 at 18:21
  • $\begingroup$ No, that was not what I meant. By the way, it is a fact that for a multiple regression model with k regressors, the t-stats for individual coefficients have (n-k-1) degrees of freedom. $\endgroup$ – ved Jul 1 '14 at 18:27
  • $\begingroup$ @ved That's exactly where my confusion comes from. My book states that in general t-statistics in a multiple regression model follow a $t(n-k)$ distribution. (So not $n-k-1$) $\endgroup$ – rbm Jul 1 '14 at 18:29
  • 1
    $\begingroup$ There is generally a lot of inconsistent notation about what "k" means in a regression model across books. Some take it to be the number of regressors, in which case the degrees of freedom is n-k-1. Some take k to be the number of regression coefficients (which is the number of regression coefficients + 1 for the intercept), so in that case the degrees of freedom would be n-k. But you have clearly specified that k is the number of regressors, so the former is correct. $\endgroup$ – ved Jul 1 '14 at 18:33
6
$\begingroup$

In my opinion there are two possible explanations here:

  1. Externally studentized residuals are based on data with one observation deleted, this may account for the loss of the single degree of freedom.

  2. There is inconsistency across books what $``k"$ actually refers to in a multiple regression model. If $k$ is the number of regressors, then the correct degrees of freedom is $n-k-1$. On the other hand, if $k$ is the number of regression coefficients (which is usually the number of regressors plus one, for the intercept), then the correct degrees of freedom is $n-k.$

Note: In general, the distribution of studentized residuals doesn't depend on whether there are dummy variables in the model or not. To be clear, let the regression model be $Y=X\beta +\epsilon$, where $X \in R^{n \times (k+1)}$, where $n$ is the number of observations and $k$ is the number of regressors. The design matrix $X$ may contain continuous variables, dummy variables and/or both. In this general framework, the internally Studentized residual is defined as
$$ r_i = \frac{e_i}{MSE(1-h_{ii})} $$ where $e_i$ is the $i^{th}$ residual, $H=(h_{ij})=(X'X)^-X'Y$ is the so-called "hat" matrix. The internally Studentized residuals do not follow a $t$-distribution, because $e_i$ and $MSE$ are not independent.

The externally Studentized residual is defined as
$$ t_i = \frac{e_i}{MSE_{(i)}(1-h_{ii})} $$ where $MSE_{(i)}$ is the mean-square error from the regression model fitted with the $i^{th}$ observation deleted. In this case, $e_i$ and $MSE_{(i)}$ are independent, and it can be shown that $t_i \sim t_{n-k-2}$, the loss of the extra one degree of freedom due to the deletion of observation $i$.

I hope this makes it clearer. So to understand the degrees of freedom in your case, you should consider the design matrix as a whole, and not break in into two parts - one with continuous predictors and one with dummy(s). Once you do that, figure out if the Studentized residuals in questions are externally or internally studentized. Then apply the above.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Sorry to say but I think I know the answer now and that this is actually incorrect. I think the answer is that because the t-statistic for the studentized residual is $\frac{e_j}{s_j\sqrt{1-h_j}}$, and because $s_j$ is computed in the regression with the dummy (and hence a zero residual for that observation), $s_j^2 \sim \frac{\chi^2(n-k-1)}{n-k-1}$, and hence the whole thing is $t(n-k-1)$ distributed. $\endgroup$ – rbm Jul 2 '14 at 13:18
  • $\begingroup$ I have made an edit, hopefully should be clearer now. $\endgroup$ – ved Jul 2 '14 at 13:37
  • $\begingroup$ Yes that's what I meant. Apparently my definition of studentized residual was then the 'externally studentized residual' (my book doesn't mention the two cases). Thanks again for your help! (One small note: in the first to last paragraph I think $n-k-2$ should be $n-k-1$; although it depends again on how you define k ;) ) $\endgroup$ – rbm Jul 2 '14 at 13:41
  • $\begingroup$ I have only one question for the answer above: Are the $e_i$ and $MSE_{(i)}$ independent in the $t_i$? If not, then how could we say that $t_i$ follows t distribution? $\endgroup$ – user117872 Jun 4 '16 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.