# Confidence set for parameter vector in linear regression

This questions is in reference to equation 3.15 in the book Elements of Statistical Learning by Tibshirani and coll.

I do understand the individual beta confidence interval estimation as provided in equation 3.14, but equation 3.15 just bowls me:

$$\beta|(\hat{\beta} -\beta)^TX^TX(\hat{\beta} -\beta)\leq \sigma ^2\chi ^2$$

What is the idea being expressed here? What is a confidence set? Can we not estimate the intervals of all the betas as per equation 3.14?

-
Please make the question self-contained. I do not have access to the book. –  varty Nov 13 '11 at 21:13
@varty: The book is freely available (legally) at the link provided by the OP. That said, I still agree the question should be made self-contained if for no other reason than that future versions, or even printings, of the text may have different equation numbering.) –  cardinal Nov 13 '11 at 21:31
Sorry this is the eqn I am referring to –  bgbgh Nov 13 '11 at 22:47
$$\beta|(\hat{\beta} -\beta)^TX^TX(\hat{\beta} -\beta)\leq \sigma ^2\chi ^2$$ –  bgbgh Nov 13 '11 at 22:48
@varty: I agree. My intent was not to be argumentative, but simply to (kindly) point out that you did have access to the book, in case you were interested. –  cardinal Nov 14 '11 at 0:24

To make things clearer recall that

$$\hat{\beta}\sim N(\beta,\hat{\sigma}^2(X^TX)^{-1}),$$

When you isolate $\beta_j$ you get that

$$\hat{\beta}_j-\beta_j\sim N(0, \sigma^2 v_j)$$

where $v_j$ are the diagonal elements of $X^TX$. We can write this alternatively as

$$\frac{\hat{\beta}_j-\beta_j}{\sqrt{v_j}}\sim N(0,\sigma),$$

which is the same as

$$\left(\frac{\hat{\beta}_j-\beta_j}{\sqrt{v_j}}\right)^2= (\hat{\beta}_j-\beta_j)(v_j)^{-1}(\hat{\beta}_j-\beta_j)\sim \sigma\chi_1^2.$$

Note that those $\beta_j$ that satisfy the condition

$$\left(\frac{\hat{\beta}_j-\beta_j}{\sqrt{v_j}}\right)^2\le \sigma^2\chi_{1,1-\alpha}^2$$

fall in the confidence interval described in the equation 3.14. Hence the confidence interval is the set in real line.

Now similarly we get $$(X^TX)^{1/2}(\hat\beta-\beta)\sim N(0, \sigma^2 I),$$

so

$$(\hat\beta-\beta)X^TX(\hat\beta-\beta)\sim \sigma^2\chi_{p+1}^2$$

where $p$ is the number of the regressors. Using the same analogy we can look for vector points $\beta\in\mathbb{R}^{p+1}$ which satisfy the condition

$$(\hat\beta-\beta)X^TX(\hat\beta-\beta)\le \sigma^2\chi_{p+1,1-\alpha}^2.$$

For $p=1$ this set will be the interior of the ellipsis.

The confidence set is used since it accounts for interactions between $\beta_i$ and $\beta_j$. Look at the scatter plot of two independent normal variables (which would be the case for orthogonal regressors with the same variance):

The circular shape is evident. Using the univariate confidence intervals the confidence set would be square, and this graph illustrates that it will actualy estimate the confidence incorrectly.

-
Thanks a lot for the response mpiktas. Here are some follow up questions Can you please elaborate on the step in which you were able to show that (betahat-beta)XtX(betahat-beta) has a chi-squared distribution? Also how does one show that for p = 1, we get an ellipse? What ellipse are you talking about? Then is that wrong to build individual tests on the isolated coefficients? –  user7413 Nov 17 '11 at 3:21
Like also the equation in the book says that its an "approximate" confidence interval, what makes it approximate. Forgive my ignorance but I wasnt able to see how you got the scatter plot, are you just plotting you independent normally distributed random variables? –  user7413 Nov 17 '11 at 3:22