Hints for exercise 7.3 from The elements of statistical learning

Question

I am stuck on problem 7.3 from the book 'The Elements of Statistical Learning'. This is the problem:

Here is my attempt to a solution for least-squares projections:

$$y_i-\hat f(x_i)=y_i-x_i^T(X_{-i}^TX_{-i})^{-1}X_{-i}^Ty,$$

where $X_{-i}$ is the matrix of training-example predictors with $i^{th}$ example (row) removed. Next I write $$X_{-i}^TX_{-i}=X^TX-x_ix_i^T$$ and $$X_{-i}^Ty_{-i}=X^Ty-x_iy_i.$$ Therefore,

$$y_i-\hat f(x_i)=y_i-x_i^T(X^TX-x_ix_i^T)^{-1}(X^Ty-x_iy_i).$$ I am not able to proceed further. It would be helpful if someone can give a hint to solve this problem.

Answer 1

Some hints: You correctly note $$ X_{-i}^TX_{-i}=X^TX-\vec{x}_i\vec{x}_i^T$$($\vec{x_i}$ is a column vector), and that you need to find $$\hat{\vec{\beta}}_{-i} = (X_{-i}^TX_{-i})^{-1}X_{-i}^T\vec{y}_{-i},$$ the estimated coefficients obtained by leaving out sample $i$. This will lead you to the new predicted value for sample $i$, $$ \hat f^{-i}(\vec{x}_i) = \vec{x}_i\hat{\vec{\beta}}_{-i}.$$ It will be useful to note that $$X_{-i}^T\vec{y}_{-i} = X^T\vec{y} - \vec{x}_iy_i,$$ and that you can use the Sherman-Morrison formula to find the inverse of $(X^TX-\vec{x}_i\vec{x}_i^T)$. Also note that $$ \vec{x}_i^T(X^TX)^{-1}\vec{x}_i = S_{ii}.$$

You can then work out what you get for $$\hat{\vec{\beta}}_{-i} = (X^TX-\vec{x}_i\vec{x}_i^T)^{-1}\left[X^Ty - \vec{x}_iy_i\right]$$ with a little algebra. Can you finish from here?