1
$\begingroup$

I'm reading the book "The Elements of Statistical Learning" (Hastie, Tibshirani, and Friedman). At page 62, they present the estimated prediction curves with the standard errors for best subset selection, ridge regression, lasso and PCR.
enter image description here
The question is: how is the standard error of a prediction error calculated? Thank you.

$\endgroup$
1
$\begingroup$

You can see that the $y$-axis is labeled CV-Error, which gives a clue.

Using cross validation, we can do the following:

for each cross validation fold train[i], test[i]:
    for each model parameter p:
        train model with parameter p on train[i]
        score model with parameter p on test[i]
        set s_{ip} = the score above

Then the line plot in the above graphs is

mean(s_{ip}) where p is fixed and i varaies

And the error bars are derived from

sd(s_{ip}) where p is fixed and i varaies

So, in words, for each parameter the mean out of fold score is calculated across all the folds, and the standard error of the out of fold score is calculated across all the folds.

$\endgroup$
  • $\begingroup$ Thank you. So the standard error bands are just [estimate-estimated.standard.deviation,estimate+estimated.standard.deviation], aren't they? $\endgroup$ – John M Jan 14 '16 at 17:56
  • $\begingroup$ I would guess that, since you are estimating the standard deviation of a mean, they are standard error bars, but that's basically correct. $\endgroup$ – Matthew Drury Jan 14 '16 at 18:00
  • $\begingroup$ Thank you. As far as I know, the variance of the sample mean is the ratio between the variance of the population and the sample size. It seems that here we don't divide the standard deviation by the sample size of the test set. Here we just take the standard deviation without dividing it. Is it because we assume to have to have k samples in a k-fold cross validation? Is my understanding correct? I promise this is my last question. Thank you again $\endgroup$ – John M Jan 14 '16 at 18:16
  • $\begingroup$ I think you're right, I spoke hastily. Thinking it through, each data point is an estimate of the mean prediction error on the out of sample fold, and the line graph is the mean of these means (whoa, mouthful). In this case, you are getting the variance of the means directly, so I think I agree with you that the standard deviation is appropriate. $\endgroup$ – Matthew Drury Jan 14 '16 at 18:39
  • $\begingroup$ I worked with this example of Hastie's book, calculating the MSE and their SE directly with the residuals obtained with CV. I calculated MSE = mean ( ( obs - pred ) ** 2 ) and the SE as = sd ( ( obs - pred ) ** 2 ) / sqrt(N) $\endgroup$ – Jesus Herranz Valera Jan 15 '16 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.