In a standard multi-variate linear regression, the distribution of the regression coefficients is a multivariate normal. We still manage to build statistical significance tests for single coefficients, totally disregarding that that coefficient is jointly normally distributed with the others. In other words why don't we do a combined testing including the effects of all coefficients?

For instance in a standard regression model, betas or the regression coefficients are jointly normal with mean equal to the true beta and the variance covariance matrix equal to $(X'X)\sigma^2$, where $\sigma$ is the variance of the errors.

  • 2
    $\begingroup$ We do not "totally disregard" that the coefficient is jointly normally distributed with the others. Just look at how the variance of each coefficient is obtained to see that. And, there are plenty of tests that take into account multiple coefficients. The standard $F$-test is one, Scheffe's method is another, Wilks' generalized variance and ellipsoids of concentration yet another. It comes down to what statistical question you are interested in. $\endgroup$ – cardinal Mar 17 '13 at 20:47
  • $\begingroup$ I guess my question is the follows, if my null hypothesis is that a particular beta is zero. How do I do it? Do I have to figure out the marginal distribution of that beta from the joint normal distribution? $\endgroup$ – gbh. Mar 17 '13 at 20:51
  • $\begingroup$ Yes, you do: It is $\sigma^2 e_i^T (X^T X)^{-1} e_i$ where $e_i$ is the elementary vector corresponding to the $i$th coefficient. $\endgroup$ – cardinal Mar 17 '13 at 20:54
  • $\begingroup$ Thanks for the note Cardinal but can you please elaborate/derive that. In particular I am reading the book by Tibshirani et al, The Elements of Statistical learning. Seems he defines the z score as z = beta/sigma*sqrt(v), where v is the corresponding diagonal element of the XtX matrix. He says that the score should have a t distribution and uses it to test for the significance. I just don't see how we get to that result. And also why that z score should have a t distribution. How do we get to that score, is that the marginal distribution? Many thanks! $\endgroup$ – gbh. Mar 17 '13 at 20:56
  • $\begingroup$ @cardinal I think with a little bit more explanation that would be an answer rather than a comment. $\endgroup$ – Peter Flom Mar 17 '13 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.