How to tell a model will do well My point is how to tell, without testing on new data, a model will predict well new data based solely on training data.
Suppose the training data is of very good quality and the selected, final cross-validated model fits the training data very well based on some efficient metric (e.g., AIC, BIC, adjusted $R^2$, Mallow's $C_P$), over all other candidate models.
Based on these assumptions, is there a metric that will tell with some degree of confidence that the model will predict unseen data well, without turning to test sets?
Suggestions would be very much appreciated.
 A: A properly implemented cross-validation or train-test split provides an unbiased estimator of performance on unseen data. This of course assumes that your train/test data is indeed representative of the future unseen data. That's often a reasonable assumption so long as you have a moderately defensible method for collecting training data (i.e. you have "good" training data), but you can't truly know that a new test cohort is drawn from the same distribution unless you actually look at it.
You couldn't, for example, train a well-tuned model of population growth in the US in 2020, and make a blanket statement that the model "will predict well on unseen data". It might do a great job of predicting population growth in the US in 2021, but probably won't do as well at predicting US population growth in 1821, or in predicting growth in Algeria in 2021. If you don't have any way to show or reasonably predict some level of similarity between your training data and the unseen test data, you can't predict your model's performance. A train/test split lets you evaluate model performance under specific circumstances, but you can't know how the model will perform under arbitrary, unknown circumstances. In fact, according to No Free Lunch, when applied to all possible circumstances, every algorithm will have the same average performance.
A: 
the training data is of very good quality

This essentially means that:

*

*the fact that the data is unseen doesn't matter

*the new unseesn data itself doesn't matter because it contains no information

Hence, any typical in-sample type of metric would be fine, R square, AIC/BIC etc.
This is a useless answer, of course. In order to get a useful answer you need to assume in what ways the unseen data is different from the training dataset.
