When we fit a machine learning model, we will use metrics like AUC or calibration error to check the model probability ranking estimate or probability estimate on the test dataset. I tried to calculate these metrics on subsets of the test dataset, such as subsets broken by month or some categorical variable. I wonder if the numbers of samples are different in these subsets, are they comparable between each other? And are they comparable to the AUC or calibration error calculated on the whole test set?
Thanks in advance!