# Statistical significance of betas in linear regression

This question is similar to another question I had recently posted but I have a follow on.

In classical linear regression, we have

$$\hat{\beta } \sim N(\beta,(X^{T}X)^{-1}\sigma^2).$$

Using this, one builds individual hypotheses of the significance of the coefficients, as done in the book by Tibshirani et al. My questions are two fold:

1) The book talks about a combined hypothesis built by proving that

$$(\hat{\beta }-\beta)^T(X^{T}X)^{-1}(\hat{\beta }-\beta) \sim \sigma^2\chi_{p+1}^2.$$

I don't see how this formula can be derived from the equation I wrote above. I do see that

$$\hat{\beta }-\beta \sim N(0,(X^{T}X)^{-1}\sigma^2).$$

How do we take the $X^TX$ matrix out and prove the above? I would be grateful if someone could outline the steps.

2) My second question is, how do we think about building the hypothesis? Do we think about building individual coefficient hypotheses or is it a good idea to view everything together? In other words, what is the difference/pros and cons of using the two different styles of hypotheses, the individual coefficient one or viewing everything together as per the above equation? Can we have an example of building a combined hypothesis? I am guessing that most statistical packages don't really take into account the correlation between different $\beta$ which is encoded in the matrix $X^TX$. Please clarify, any help will be much appreciated.

-
Presumably, in the second formula, $\le$ is a typo for $\sim$ and $p+1$ is the number of parameters (including the constant), right? –  whuber Nov 18 '11 at 18:48
Yes, and Yes. Can you make sense of it though, how does the second formula come? –  bgbgh Nov 18 '11 at 19:42
The left hand side in (1) is a sum of squares of $p+1$ normal variables. By definition, a chi-squared distribution describes a sum of squares of $p+1$ standard normal variables. –  whuber Nov 18 '11 at 20:30
Factor it: because it is symmetric positive-definite, you can write it as $X^TX = UU^T$ for an invertible $p+1$ by $p+1$ matrix $U$. –  whuber Nov 18 '11 at 20:39
You would benefit greatly from working an actual problem. Why don't you fit a line through the points $(0,2),(3,4),(4,8),(7,10)$. (I chose this for the easy arithmetic.) You should compute $U = \{\{2,0\},\{7,5\}\}$. Then you might see where the correlation appears. –  whuber Nov 18 '11 at 21:37
1) I do not have the book with me so I cannot check the original, but there is a typo in the first formula as given in that it should be $$(\hat\beta-\beta)^T(X^TX)(\hat\beta-\beta)\sim \sigma^2\chi^2_{p+1}$$ with no inverse for the $(X^T X)$ matrix.
This is a consequence of the following: if $x\sim\mathcal{N}_p(0,\Sigma)$, then $$Ax\sim\mathcal{N}_q(0,A\Sigma A^T)$$ for any $(q,p)$ matrix $A$. Thus, taking one symmetric version of the square root of $(X^TX)$, i.e. $V$ such that $V^TV=(X^TX)$ and $V(X^TX)^{-1}V^T=I_{p+1}$, using for instance the eigenbasis and eigenvalues, you get that $$V(\hat\beta-\beta)\sim\mathcal{N}_{p+1}(0,\sigma^2I_{p+1})$$ and $$(\hat\beta-\beta)^TV^TV(\hat\beta-\beta)=(\hat\beta-\beta)^T(X^TX)(\hat\beta-\beta)\sim \sigma^2\chi^2_{p+1}.$$
2) again, I do not have the book so cannot guess what the author mean by "building an hypothesis". The natural approach is to have an exogenous question about the significance of one group of variables and to test it by the corresponding chi-square test, using the corresponding submatrix of $(X^TX)^{-1}$. For instance, testing for $\beta_1=\beta_2=0$ leads to $$(\hat\beta_{1:2}-\beta_{1:2})^T\left[(X^TX)^{-1}_{1:2,1:2}\right]^{-1}(\hat\beta_{1:2} -\beta_{1:2})\sim \sigma^2\chi^2_{2}.$$