I'm using a k-fold (10-fold) cross-validation while building a model. I'm only using it to get an estimate of the out-of-sample error, not to pick a model from candidates.

For example, if I have 30 points, I use 27 points to train a regression model, and record the (squared) prediction error of the remaining 3: (yPrediction-yActual)^2. I repeat this 10 times until all 30 points have been used as the 3 test points once. For each model I take the mean of the 3 prediction errors as an estimate for the error of that particular model.

These means are then averaged for an overall estimate of out of sample error (over all 10 models built). Finally, I use all 30 points at once to build a final model. From my understanding the square root of the mean of squared prediction errors is the RMSECV. The next step is to use this to give a prediction interval on new points. For example, the new point will be 10+/- PI, where PI is a quantity based on the error estimate from CV and some parameter for confidence (e.g. 95%).

I think this is a common practice in Machine Learning, but I'm struggling to figure out how to go from CV to prediction intervals. Does anyone have a good book or paper that explains this in layman's terms? I'm not really from a ML or stats background. PS: I'm using MATLAB.

Usage example (Added for clarity of question):

For example, lets say I build a 1-Variable linear regression as my model using training data for points [1,3,7,10,12,15]. I test my model with LOOCV while building it, and my CV RSME is 2.4. I test a few more points and the EOut RSME is about 2.3 (very similar to the validation error). I'm satisfied with this, and want to use the model to predict new responses.

So, I give my model an input of 5 and the predicted response is 40. However, I know from the non-zero RSME, my model has error. I've quantified it. Therefore, I'd rather tell the user the response is somewhere near 40 +/- some tolerance (with X% certainty). Something like 40 +/- 3 (with 95% certainty), instead of just 40.

Is it possible to do this with the CV RMSE, or do I need more information? Can that information (e.g. distribution) be assumed?

  • 1
    $\begingroup$ Look up "Conformal prediction". $\endgroup$ – Emre Jun 19 '14 at 20:39

what do you mean by prediction interval on new points ? In general if your CV error is less then you proceed further to verify it on test data set. If the result on test data set is also good then use the parameters for now unknown data points otherwise improve the model.

A general overview is explained here: http://www.holehouse.org/mlclass/10_Advice_for_applying_machine_learning.html

| cite | improve this answer | |
  • 1
    $\begingroup$ Hello, I edited the question to be a bit more clear of what I'm looking for. I hope that helps. Thanks, for the link BTW. $\endgroup$ – user3712310 Jun 9 '14 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.