Background: I'm working on a ML project to predict a continuous target and am comparing different models using nested cross-validation, where I don't have access to the test set for which my model will be judged. My data size is on the order of $100,000 \text{ samples}\ \times 10 \text { features}$.

I thought nested cross-validation would be appropriate because in the book Python Machine Learning by Raschka, he refers to a paper S. Varma and R. Simon. Bias in Error Estimation When Using Cross-validation for Model Selection. BMC bioinformatics, 7(1):91, 2006)., where he says the authors concluded that, "the true error of the estimate is almost unbiased relative to the test set when nested cross-validation is used."

I have decent experience with programming and probability theory, but my basic stats background has some holes. So my question is, is there any general way to quantify the expected uncertainty on unseen data for the models I create through nested cross-validation? Something like confidence intervals or standard error intervals in addition to predictions?

I plan to try many different types of models, like for example, Lasso, Ridge, ElasticNet, GradientBoostingRegressor, XGBRegressor, etc. Is quantifying uncertainty model specific or is there some general way for which I may quantify how accurate I believe my final models' predictions will be on new data?

I hope this question makes sense.. please let me know if it does not.

  • 2
    $\begingroup$ This is the kind of thing bootstrapping is made for. Once you have your nested-cv procedure planned out, you can run it over many bootstrap samples of your data. Each time you run on a bootstrap sample, you'll get predictions for your out-of-fold test data points. If you keep track of these predictions for each bootstrap sample, you can aggregate together all these estimates across the bootstrap samples, and get estimates of prediction variance. $\endgroup$ – Matthew Drury Dec 29 '17 at 23:50
  • $\begingroup$ Thanks for your comment. Ok so I think I may understand now. I'll choose a model based on nested cross-validation with the entire data set beforehand, then I'll use bootstrapped samples to get a distribution of the performance metric, such as RMSE, and then calculate a confidence interval on that distribution as in machinelearningmastery.com/… $\endgroup$ – Austin Dec 30 '17 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.