I'm using a k-fold (10-fold) cross-validation while building a model. I'm only using it to get an estimate of the out-of-sample error, not to pick a model from candidates.
For example, if I have 30 points, I use 27 points to train a regression model, and record the (squared) prediction error of the remaining 3: (yPrediction-yActual)^2. I repeat this 10 times until all 30 points have been used as the 3 test points once. For each model I take the mean of the 3 prediction errors as an estimate for the error of that particular model.
These means are then averaged for an overall estimate of out of sample error (over all 10 models built). Finally, I use all 30 points at once to build a final model. From my understanding the square root of the mean of squared prediction errors is the RMSECV. The next step is to use this to give a prediction interval on new points. For example, the new point will be 10+/- PI, where PI is a quantity based on the error estimate from CV and some parameter for confidence (e.g. 95%).
I think this is a common practice in Machine Learning, but I'm struggling to figure out how to go from CV to prediction intervals. Does anyone have a good book or paper that explains this in layman's terms? I'm not really from a ML or stats background. PS: I'm using MATLAB.
Usage example (Added for clarity of question):
For example, lets say I build a 1-Variable linear regression as my model using training data for points [1,3,7,10,12,15]. I test my model with LOOCV while building it, and my CV RSME is 2.4. I test a few more points and the EOut RSME is about 2.3 (very similar to the validation error). I'm satisfied with this, and want to use the model to predict new responses.
So, I give my model an input of 5 and the predicted response is 40. However, I know from the non-zero RSME, my model has error. I've quantified it. Therefore, I'd rather tell the user the response is somewhere near 40 +/- some tolerance (with X% certainty). Something like 40 +/- 3 (with 95% certainty), instead of just 40.
Is it possible to do this with the CV RMSE, or do I need more information? Can that information (e.g. distribution) be assumed?