How to verify if a prediction performance improvement is significant better? I have a model M1 that achieved the predictive score (accuracy / AUC / F1 ...) of s1 in the test dataset.
I developed a new model M2 that achieved the predictive score of s2 on the same testset. Suppose that s2 > s1.
Is there any test to confirm that M2 is actually better than M1, or it is just the result of randomness in the models?
 A: A few possibilities come to mind.
If you have multiple datasets, you could use a Diebold-Mariano test. The tag wiki contains information, as well as pointers to the original and a follow-up publication. The DM test is very commonly applied in the time series forecasting community, but there is nothing specifically "time-seriesey" about it.
If you have only a single dataset, then you could in principle bootstrap your model fits for both models and assess whether one improves on the other in (say) 95% of cases.
If you do this on the same test set in each replicate (only bootstrapping the training data), then all you can conclude is whether one model is better on this particular test set, so the bare minimum I would do would be to wrap the exercise in a cross-validation step. And even so, applying this machinery on a single dataset would be very prone to just making you very confident that an overfitting model is better than it actually is.
A: Dietterich (1998) proposes a 5x2 cross validation followed by a t-test to identify better models. When comparing models across multiple datasets, Demsar (2006) suggests the use of Wilcoxon Signed Rank test for comparing two models and Friedman test when comparing more than two models.
