An Introduction to Statistical Learning by James et al. defines the leave-one-out cross-validation estimate for least-squares as:

$$CV_{(n)}=\frac{1}{n}\sum_{i=1}^n \biggr(\frac{y_i-\hat y_i}{1-h_i}\biggr)^2$$

with $\hat y_i$ being the $i$th fitted value form the original least squares fit, and $h_i$ being the leverage defined by $h_i=\frac{1}{n} +\frac{(x_i- \bar x)^2}{\sum_{i'=1}^n (x_{i'}-\bar x)^2 }$. I've been able to find and understand the proof using matrices for this (https://robjhyndman.com/hyndsight/loocv-linear-models/), but I'm having trouble translating it to the case of regression through the origin in a meaningful way.

For the model $Y=\beta X+\epsilon$ I've found the estimate of $\beta$ to be $$\hat \beta=\frac{\sum_{k=1}^n x_{k}y_k}{\sum_{k=1}^n x_{k}^2}$$

and the variance of the residual error would be $$\frac{1}{n}\sum_{i=1}^n (y_i-\hat y_i^{-i})^2$$ where $y_i^{-i}$ is the model output when $(x_i,y_i)$ are excluded from the training data. I just can't figure out what the next step would be to apply this for least-squares regression through the origin.

  • 2
    $\begingroup$ Welcome to CV! In your second equation ($\hat{\beta} = \dots$), is seems as though you are using all data to estimate $\hat{\beta}$. If you want to use LOOCV, you should omit the $i$-th observation from the estimation at each step $i$. $\endgroup$ – Frans Rodenburg Oct 29 '18 at 3:42
  • $\begingroup$ @Frans Rodenburg Thank you! I should have clarified my definition a bit better, that estimate of $\hat \beta$ corresponds to the model where the entire sample is used, with response $ \hat \y_{i} $. Am I correct in assuming that the $\hat \beta^{-i}$ parameter for model with the $i$th observation removed would be the same summation, expect split into two parts with one summing from $k=1$ to $i-1$ and the other from $k=i+1$ to $n$? $\endgroup$ – sk13 Oct 29 '18 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.