Background: I'm working on a ML project to predict a continuous target and am comparing different models using nested cross-validation, where I don't have access to the test set for which my model will be judged. My data size is on the order of $100,000 \text{ samples}\ \times 10 \text { features}$.
I thought nested cross-validation would be appropriate because in the book Python Machine Learning by Raschka, he refers to a paper S. Varma and R. Simon. Bias in Error Estimation When Using Cross-validation for Model Selection. BMC bioinformatics, 7(1):91, 2006)., where he says the authors concluded that, "the true error of the estimate is almost unbiased relative to the test set when nested cross-validation is used."
I have decent experience with programming and probability theory, but my basic stats background has some holes. So my question is, is there any general way to quantify the expected uncertainty on unseen data for the models I create through nested cross-validation? Something like confidence intervals or standard error intervals in addition to predictions?
I plan to try many different types of models, like for example, Lasso, Ridge, ElasticNet, GradientBoostingRegressor, XGBRegressor, etc. Is quantifying uncertainty model specific or is there some general way for which I may quantify how accurate I believe my final models' predictions will be on new data?
I hope this question makes sense.. please let me know if it does not.
