Timeline for Nested cross-validation and quantifying uncertainty
Current License: CC BY-SA 3.0
3 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 30, 2017 at 19:01 | comment | added | Austin | Thanks for your comment. Ok so I think I may understand now. I'll choose a model based on nested cross-validation with the entire data set beforehand, then I'll use bootstrapped samples to get a distribution of the performance metric, such as RMSE, and then calculate a confidence interval on that distribution as in machinelearningmastery.com/… | |
| Dec 29, 2017 at 23:50 | comment | added | Matthew Drury | This is the kind of thing bootstrapping is made for. Once you have your nested-cv procedure planned out, you can run it over many bootstrap samples of your data. Each time you run on a bootstrap sample, you'll get predictions for your out-of-fold test data points. If you keep track of these predictions for each bootstrap sample, you can aggregate together all these estimates across the bootstrap samples, and get estimates of prediction variance. | |
| Dec 29, 2017 at 23:28 | history | asked | Austin | CC BY-SA 3.0 |