How does train-validation-test procedure deals with the sampling error of the accuracy measure?

Question

Let's consider a standard model selection procedure:

1. We have N different untrained models (for example linear regression, neural network, decision tree and so on).
2. We use a data set A to train each of these models.
3. We use a data set B to validate each of the training models.
4. We choose a model that gives the best score on the data set B.
5. We do a final test of the choses model using a data set C.

It looks to me that this procedure does not take into account the fact that our measure of accuracy (for example mean squared deviation) is stochastic (because we have only sample mean of the measure and our samples (A, B and C) are limited).

In particular, it might be the case that one of the trained models has the best score on the data set B (validation data set) just by chance (because of the stochastic sampling error). For example would we have another validation data set (data set B2), we would see that it is not the best model anymore.

As far as I understand, this is exactly we use the data set C (test data set). The model that is the best on the data set C (validation data set), should also show good / comparable results on a data set C. However, where is the guaranty that the considered model will show good results on the data set C also "just by chance". In other words, it might be the case the the model "just got lucky" to show good results on 2 data sets. Isn't the case that 2 good results is not enough? Shouldn't we have, let's say 10 "good results", to be confident?

The other side of the problem is that during our validation we could through away a good model just because it was "unlucky" (because of the sampling error). It gave bad results on the validation data set B but it would give good results on the test results C (but we never tried it because it did not pass the validation test).

So, my question is that if there are some extensions of the train-validation-test procedure that deals with the fact that the accuracy measure is stochastic because of the sampling error?

Answer 1

TL:DR on average it works out.

The idea is that you use the same data for each and every model so that they are completely comparable. As for whether a particular portion of the data, say A, works better for one model, that is out of your control. As long as you create the partitions randomly, this should ensure that there are no biases in any one partition for any one model (on average). If you are worried about this you could perform repeated partitions of your data, each time selecting randomly. If one model works better on the whole data regardless of partitioning then that is a sign that this model is better suited for the data.