How to determine if one predictive model is statistically significantly better than another one? I have a data set, two competing predictive models (regressions) and I need to decide which predictive model is better. Let us also assume that I have a measure of accuracy (for example mean squared deviation).
As the first step, I can randomly split my data set into two equal parts. I use the first part to train each of the two models and then I evaluate the two trained models using the second data set. In that way I get two out-of-sample errors. Then I could say that the model corresponding to the smaller error is better. However, how can I be sure that the results are statistically significant? For example, it can be the case that the first model wins for one split of the data and looses for another split.
I thought that, as a workaround, I could take all the deviations between the targets and the corresponding predictions and then resample from these deviations. Then I check how frequently one model is better / worse than another one. However, this way I do not count for the randomness determined by the limited size of the training set. In other words, it might be the case that model 1 is statistically significantly better than the model 2 if both models are trained on the first 100 randomly selected data points but if we use other 100 data points, model 2 can be better than model 1.
I guess that the main cause of my problem is that the measure of accuracy is random because of two reasons: (1) the limited sizes of the evaluation set and (2) limited size of the training set. So, I need to have a way to treat this randomness properly.
 A: You are right that your validation/verification results are subject to (at least) these two sources of variance:

the measure of accuracy is random because of two reasons: (1) the limited sizes of the evaluation set and (2) limited size of the training set.

Variance (2) is commonly called model instability.
Practically speaking, there are 3 situations:


*

*Var (1) >> Var (2): you can use Var (1) as approximation for the total variance.

*Var (1) << Var (2): you can use Var (2) as approximation for the total variance.
However, you may want to reconsider the meaning of this finding: the predictions of the model in question is unstable wrt. exchanging some training cases for other training cases from the same population. Maybe you should improve your training first in order to arrive at a stable models.

*variance (1) and (2) are in the same order of magnitude. Both of them need to be taken into account for the decision.
Again, think whether the observed level of instability is acceptable for your application.
You can measure Var (2) rather directly with iterated/repeated cross validation (see e.g. our paper: Beleites, C. & Salzer, R. Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 2008, 390, 1261-1271), but also out-of-bootstrap allows you to separate the two variance sources.

When formulating the test, keep in mind that Var (2) is likely or even expected to vary between models. 
