Cross-validation error of ridge regression - Cross Validated most recent 30 from stats.stackexchange.com 2019-07-23T04:56:51Z https://stats.stackexchange.com/feeds/question/409978 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/409978 0 Cross-validation error of ridge regression Mr.Robot https://stats.stackexchange.com/users/191779 2019-05-24T18:54:36Z 2019-05-24T18:54:36Z <h2>Problem</h2> <p>In order to find the optimal parameter <span class="math-container">$\lambda$</span>, each individual observation is taken out from design matrix <span class="math-container">$\mathbf{X}$</span> and solves <span class="math-container">$$\text{minimize}_{\beta} \frac{1}{2}\Vert \mathbf{y}_{-k}-\mathbf{X}_{-k}\beta\Vert_2^2+\frac{1}{2}\lambda\Vert\beta\Vert_2^2$$</span> where <span class="math-container">$\mathbf{y}_{-k}$</span> and <span class="math-container">$\mathbf{X}_{-k}$</span> means the vector and design matrix with <span class="math-container">$k$</span>-th response and sample taken out.</p> <p>Then the cross validation error is naturally defined as <span class="math-container">$$C(\lambda)=\frac{1}{n}\sum_{k=1}^m[y_k-\mathbf{x}_k^T\beta_k]^2$$</span> However, this error is <strong>claimed</strong> to be equivalent to the following with SVD of <span class="math-container">$\mathbf{X}$</span> <span class="math-container">$$C(\lambda)=\frac{1}{n} \sum_{k=1}^{n}\left[\frac{y_{k}-\sum_{j=1}^{r} \mathbf{u}_{k j} \mathbf{u}_j^T\mathbf{y}\left(\frac{\sigma_{j}^{2}}{\sigma_{j}^{2}+\lambda}\right)}{1-\sum_{j=1}^{r} \mathbf{u}_{k j}^{2}\left(\frac{\sigma_{j}^{2}}{\sigma_{j}^{2}+\lambda}\right)}\right]^{2}$$</span> which I do not quite see why it is the case especially I do not understand how the denominator comes into play.</p> <p>Could someone help me, thank you in advance!</p>