I have a data set, two competing predictive models (regressions) and I need to decide which predictive model is better. Let us also assume that I have a measure of accuracy (for example mean squared deviation).
As the first step, I can randomly split my data set into two equal parts. I use the first part to train each of the two models and then I evaluate the two trained models using the second data set. In that way I get two out-of-sample errors. Then I could say that the model corresponding to the smaller error is better. However, how can I be sure that the results are statistically significant? For example, it can be the case that the first model wins for one split of the data and looses for another split.
I thought that, as a workaround, I could take all the deviations between the targets and the corresponding predictions and then resample from these deviations. Then I check how frequently one model is better / worse than another one. However, this way I do not count for the randomness determined by the limited size of the training set. In other words, it might be the case that model 1 is statistically significantly better than the model 2 if both models are trained on the first 100 randomly selected data points but if we use other 100 data points, model 2 can be better than model 1.
I guess that the main cause of my problem is that the measure of accuracy is random because of two reasons: (1) the limited sizes of the evaluation set and (2) limited size of the training set. So, I need to have a way to treat this randomness properly.