Let's consider a standard model selection procedure:
- We have N different untrained models (for example linear regression, neural network, decision tree and so on).
- We use a data set A to train each of these models.
- We use a data set B to validate each of the training models.
- We choose a model that gives the best score on the data set B.
- We do a final test of the choses model using a data set C.
It looks to me that this procedure does not take into account the fact that our measure of accuracy (for example mean squared deviation) is stochastic (because we have only sample mean of the measure and our samples (A, B and C) are limited).
In particular, it might be the case that one of the trained models has the best score on the data set B (validation data set) just by chance (because of the stochastic sampling error). For example would we have another validation data set (data set B2), we would see that it is not the best model anymore.
As far as I understand, this is exactly we use the data set C (test data set). The model that is the best on the data set C (validation data set), should also show good / comparable results on a data set C. However, where is the guaranty that the considered model will show good results on the data set C also "just by chance". In other words, it might be the case the the model "just got lucky" to show good results on 2 data sets. Isn't the case that 2 good results is not enough? Shouldn't we have, let's say 10 "good results", to be confident?
The other side of the problem is that during our validation we could through away a good model just because it was "unlucky" (because of the sampling error). It gave bad results on the validation data set B but it would give good results on the test results C (but we never tried it because it did not pass the validation test).
So, my question is that if there are some extensions of the train-validation-test procedure that deals with the fact that the accuracy measure is stochastic because of the sampling error?