In order to find the optimal parameter $\lambda$, each individual observation is taken out from design matrix $\mathbf{X}$ and solves $$ \text{minimize}_{\beta} \frac{1}{2}\Vert \mathbf{y}_{-k}-\mathbf{X}_{-k}\beta\Vert_2^2+\frac{1}{2}\lambda\Vert\beta\Vert_2^2 $$ where $\mathbf{y}_{-k}$ and $\mathbf{X}_{-k}$ means the vector and design matrix with $k$-th response and sample taken out.

Then the cross validation error is naturally defined as $$ C(\lambda)=\frac{1}{n}\sum_{k=1}^m[y_k-\mathbf{x}_k^T\beta_k]^2 $$ However, this error is claimed to be equivalent to the following with SVD of $\mathbf{X}$ $$ C(\lambda)=\frac{1}{n} \sum_{k=1}^{n}\left[\frac{y_{k}-\sum_{j=1}^{r} \mathbf{u}_{k j} \mathbf{u}_j^T\mathbf{y}\left(\frac{\sigma_{j}^{2}}{\sigma_{j}^{2}+\lambda}\right)}{1-\sum_{j=1}^{r} \mathbf{u}_{k j}^{2}\left(\frac{\sigma_{j}^{2}}{\sigma_{j}^{2}+\lambda}\right)}\right]^{2} $$ which I do not quite see why it is the case especially I do not understand how the denominator comes into play.

Could someone help me, thank you in advance!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.