2
$\begingroup$

I'm working on Exercise 3.2 from Elements of Statistical Learning. It asks to find a $95\%$ confidence interval for a linear regression prediction (ordinary least squares are used) using two different methods.

In the first part we're looking for the confidence interval for a single prediction $x_0^T \hat{\beta}$, as far as I understand it's basically

$$x_0^T \hat{\beta} \pm z_{0.975} \cdot \hat{\sigma} \sqrt{x_0^T \left(\mathbf{X}^T \mathbf{X}\right) x_0},$$

where $\mathbf{X} \in \mathbb{R}^{N \times (p + 1)}$ - design matrix, $z_{0.975} \approx 1.96$ quantile of standard normal distribution and

$$\hat{\sigma} = \frac{1}{N - p - 1} \sum_{i = 1}^N \left( y_i - \hat{y}_i \right)^2$$

In the second part, we're looking for a confidence interval generated by a confidence set for the whole vector $\beta$:

$$C_\beta = \left\{ \beta: \; f(\beta) = (\beta - \hat{\beta})^T \mathbf{X}^T \mathbf{X} (\beta - \hat{\beta}) \leq \hat{\sigma} \chi_{p+1}^{2 \; (0.975)}\right\},$$

where $\chi_{p+1}^{2 \; (0.975)}$ is a $0.975$ quantile of $\chi_{p + 1}^2$. Here I'm having troubles. First of all, how to calculate confidence interval based on $C_\beta$? My guess was that it's equivalent to finding the following:

$$\min_{\beta \in C_\beta} x_0^T \beta \quad \text{and} \quad \min_{\beta \in C_\beta} x_0^T \beta$$

which at the same time is equivalent to

$$\begin{cases} \alpha x_0 = \nabla f (\beta) \\ \beta \in \partial C_\beta \end{cases}$$

for a fixed $x_0$ (here I used Lagrange multipliers method for solving an optimization problem). I've solved this and got the following confidence interval for $x_0^T \hat{\beta}$:

$$x_0^T \hat{\beta} \pm \hat{\sigma} \sqrt{\chi_{p+1}^{2 \; (0.975)} x_0^T \left(\mathbf{X}^T \mathbf{X}\right) x_0}$$

Is this right? My main concern is that it's just a $\chi^2$ correction of the first confidence interval, where we used estimate of $\sigma$ but not its true value, though is it indeed a confidence interval generated by $C_\beta$? And if it's not, could someone help with understanding what it actually is?

Thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.