I read that in a regression with $k$ regressors, the t-statistic corresponding to a certain coefficient follows a $t(n-k)$ distribution. However, later on I read that studentized residuals follow a $t(n-k-1)$ distribution. How could it be that the extra degree of freedom is lost: isn't this also just based on a regular t-statistic?
$\begingroup$
$\endgroup$
-
2$\begingroup$ Do you mean externally studentized residuals? These are calculated by deleting one observation, which would explain a loss of 1 degree of freedom. $\endgroup$ – ved Jul 1 '14 at 18:03
-
$\begingroup$ @ved Thanks for your reply. To be honest I'm not entirely sure what you mean with "externally" in this context. The studentized residual that I mean is the t-statistic for $\hat{\gamma}$ that can be obtained from the regression $y=X\beta + \gamma D_j$. Do you mean that the fact that there is a dummy in the regression somehow implies that the degree of freedom is lost? $\endgroup$ – rbm Jul 1 '14 at 18:21
-
$\begingroup$ No, that was not what I meant. By the way, it is a fact that for a multiple regression model with k regressors, the t-stats for individual coefficients have (n-k-1) degrees of freedom. $\endgroup$ – ved Jul 1 '14 at 18:27
-
$\begingroup$ @ved That's exactly where my confusion comes from. My book states that in general t-statistics in a multiple regression model follow a $t(n-k)$ distribution. (So not $n-k-1$) $\endgroup$ – rbm Jul 1 '14 at 18:29
-
1$\begingroup$ There is generally a lot of inconsistent notation about what "k" means in a regression model across books. Some take it to be the number of regressors, in which case the degrees of freedom is n-k-1. Some take k to be the number of regression coefficients (which is the number of regression coefficients + 1 for the intercept), so in that case the degrees of freedom would be n-k. But you have clearly specified that k is the number of regressors, so the former is correct. $\endgroup$ – ved Jul 1 '14 at 18:33
$\begingroup$
$\endgroup$
In my opinion there are two possible explanations here:
Externally studentized residuals are based on data with one observation deleted, this may account for the loss of the single degree of freedom.
There is inconsistency across books what $``k"$ actually refers to in a multiple regression model. If $k$ is the number of regressors, then the correct degrees of freedom is $n-k-1$. On the other hand, if $k$ is the number of regression coefficients (which is usually the number of regressors plus one, for the intercept), then the correct degrees of freedom is $n-k.$
Note: In general, the distribution of studentized residuals doesn't depend on whether there are dummy variables in the model or not. To be clear, let the regression model be $Y=X\beta +\epsilon$, where $X \in R^{n \times (k+1)}$, where $n$ is the number of observations and $k$ is the number of regressors. The design matrix $X$ may contain continuous variables, dummy variables and/or both. In this general framework, the internally Studentized residual is defined as
$$
r_i = \frac{e_i}{MSE(1-h_{ii})}
$$
where $e_i$ is the $i^{th}$ residual, $H=(h_{ij})=(X'X)^-X'Y$ is the so-called "hat" matrix. The internally Studentized residuals do not follow a $t$-distribution, because $e_i$ and $MSE$ are not independent.
The externally Studentized residual is defined as
$$
t_i = \frac{e_i}{MSE_{(i)}(1-h_{ii})}
$$
where $MSE_{(i)}$ is the mean-square error from the regression model fitted with the $i^{th}$ observation deleted. In this case, $e_i$ and $MSE_{(i)}$ are independent, and it can be shown that $t_i \sim t_{n-k-2}$, the loss of the extra one degree of freedom due to the deletion of observation $i$.
I hope this makes it clearer. So to understand the degrees of freedom in your case, you should consider the design matrix as a whole, and not break in into two parts - one with continuous predictors and one with dummy(s). Once you do that, figure out if the Studentized residuals in questions are externally or internally studentized. Then apply the above.
-
$\begingroup$ Sorry to say but I think I know the answer now and that this is actually incorrect. I think the answer is that because the t-statistic for the studentized residual is $\frac{e_j}{s_j\sqrt{1-h_j}}$, and because $s_j$ is computed in the regression with the dummy (and hence a zero residual for that observation), $s_j^2 \sim \frac{\chi^2(n-k-1)}{n-k-1}$, and hence the whole thing is $t(n-k-1)$ distributed. $\endgroup$ – rbm Jul 2 '14 at 13:18
-
$\begingroup$ I have made an edit, hopefully should be clearer now. $\endgroup$ – ved Jul 2 '14 at 13:37
-
$\begingroup$ Yes that's what I meant. Apparently my definition of studentized residual was then the 'externally studentized residual' (my book doesn't mention the two cases). Thanks again for your help! (One small note: in the first to last paragraph I think $n-k-2$ should be $n-k-1$; although it depends again on how you define k ;) ) $\endgroup$ – rbm Jul 2 '14 at 13:41
-
$\begingroup$ I have only one question for the answer above: Are the $e_i$ and $MSE_{(i)}$ independent in the $t_i$? If not, then how could we say that $t_i$ follows t distribution? $\endgroup$ – user117872 Jun 4 '16 at 4:16
