# How to verify if a prediction performance improvement is significant better?

I have a model M1 that achieved the predictive score (accuracy / AUC / F1 ...) of s1 in the test dataset.

I developed a new model M2 that achieved the predictive score of s2 on the same testset. Suppose that s2 > s1.

Is there any test to confirm that M2 is actually better than M1, or it is just the result of randomness in the models?

• Is there any randomness involved in building your models or are they completely deterministic? – user2974951 Feb 11 at 8:45
• @user2974951randomness might come from data sampling as well - so the question here is that is there a kind of test that can say a model is actually better than other, like t-test in statistics. – mommomonthewind Feb 11 at 8:55
• No, this is not really done. If your models are random in nature then you could build multiple models and check their averages / variances, you could even use a test to check for significance, but this is dubious. – user2974951 Feb 11 at 8:58