I am stuck on problem 7.3 from the book 'The Elements of Statistical Learning'. This is the problem: enter image description here

Here is my attempt to a solution for least-squares projections:

$$y_i-\hat f(x_i)=y_i-x_i^T(X_{-i}^TX_{-i})^{-1}X_{-i}^Ty,$$

where $X_{-i}$ is the matrix of training-example predictors with $i^{th}$ example (row) removed. Next I write $$X_{-i}^TX_{-i}=X^TX-x_ix_i^T$$ and $$X_{-i}^Ty_{-i}=X^Ty-x_iy_i.$$ Therefore,

$$y_i-\hat f(x_i)=y_i-x_i^T(X^TX-x_ix_i^T)^{-1}(X^Ty-x_iy_i).$$ I am not able to proceed further. It would be helpful if someone can give a hint to solve this problem.

  • 1
    $\begingroup$ Use the self study tag. $\endgroup$ – Michael R. Chernick Oct 7 '17 at 18:42
  • $\begingroup$ I have used this tag. $\endgroup$ – Abhinav Gupta Oct 7 '17 at 18:48

Some hints: You correctly note $$ X_{-i}^TX_{-i}=X^TX-\vec{x}_i\vec{x}_i^T$$($\vec{x_i}$ is a column vector), and that you need to find $$\hat{\vec{\beta}}_{-i} = (X_{-i}^TX_{-i})^{-1}X_{-i}^T\vec{y}_{-i},$$ the estimated coefficients obtained by leaving out sample $i$. This will lead you to the new predicted value for sample $i$, $$ \hat f^{-i}(\vec{x}_i) = \vec{x}_i\hat{\vec{\beta}}_{-i}.$$ It will be useful to note that $$X_{-i}^T\vec{y}_{-i} = X^T\vec{y} - \vec{x}_iy_i,$$ and that you can use the Sherman-Morrison formula to find the inverse of $(X^TX-\vec{x}_i\vec{x}_i^T)$. Also note that $$ \vec{x}_i^T(X^TX)^{-1}\vec{x}_i = S_{ii}.$$

You can then work out what you get for $$\hat{\vec{\beta}}_{-i} = (X^TX-\vec{x}_i\vec{x}_i^T)^{-1}\left[X^Ty - \vec{x}_iy_i\right]$$ with a little algebra. Can you finish from here?

| cite | improve this answer | |
  • $\begingroup$ Let me try it out in some time. I will let you know then. it seems like I would be able to solve this using Sherman-Morrison formula. Thanks a lot! $\endgroup$ – Abhinav Gupta Oct 7 '17 at 22:04
  • $\begingroup$ It worked. I was able to solve it using Sherman-Morrison formula for $(X^TX-x_ix_i^T)^{-1}$ $\endgroup$ – Abhinav Gupta Oct 8 '17 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.