When we fit a machine learning model, we will use metrics like AUC or calibration error to check the model probability ranking estimate or probability estimate on the test dataset. I tried to calculate these metrics on subsets of the test dataset, such as subsets broken by month or some categorical variable. I wonder if the numbers of samples are different in these subsets, are they comparable between each other? And are they comparable to the AUC or calibration error calculated on the whole test set?

Thanks in advance!

  • $\begingroup$ At least in the case of AUC, this statistic reflects the expectation of a particular event, so it is comparable to other expectations computed with varying sample sizes; naturally, these statistics will have lower variance when the sample size is larger, so it's worthwhile to account for that. I'm not familiar with the "calibration error" statistic. $\endgroup$ – Sycorax Feb 9 at 22:43
  • $\begingroup$ @Sycorax Thanks! but how to account for different sample size? Lets say auc for subset Jan has 1/10 of samples as subset Feb... usually AUC for Feb is higher than AUC for Jan, but can i say the model fits better for Feb data than Jan data? $\endgroup$ – NewYear2021 Feb 10 at 9:34
  • $\begingroup$ You'll need to do a statistical hypothesis test to compare the two AUCs. $\endgroup$ – Sycorax Feb 10 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.