5
$\begingroup$

I have a model M1 that achieved the predictive score (accuracy / AUC / F1 ...) of s1 in the test dataset.

I developed a new model M2 that achieved the predictive score of s2 on the same testset. Suppose that s2 > s1.

Is there any test to confirm that M2 is actually better than M1, or it is just the result of randomness in the models?

$\endgroup$
  • $\begingroup$ Is there any randomness involved in building your models or are they completely deterministic? $\endgroup$ – user2974951 Feb 11 at 8:45
  • $\begingroup$ @user2974951randomness might come from data sampling as well - so the question here is that is there a kind of test that can say a model is actually better than other, like t-test in statistics. $\endgroup$ – mommomonthewind Feb 11 at 8:55
  • $\begingroup$ No, this is not really done. If your models are random in nature then you could build multiple models and check their averages / variances, you could even use a test to check for significance, but this is dubious. $\endgroup$ – user2974951 Feb 11 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.