I have a binary classification set with about 900 samples. I first use kfold cross validation and AUC of the ROC curve to determine which classifiers perform best. Once I've done that, I use a leave-one-group out approach for additional metrics. In my dataset, the data is collected among 32 groups. Groups within my set are uneven. It can happen for example that one group will have 32 class0 samples, and 0 class1 samples. One result of classification could then be
Class 0 [30 2] Class 1 [0 0]
However, in this case, tpr = TP/(TP+FN) = 0 / (0+0) and I get a NaN. My question is what should I do in this case? My plan was to use AUC to compare classifiers using leave-one-group-out approach, but this will obviously not work because of what I've just mentioned. Ive thought of ignoring groups like this, but there are more than 1 and it feels wrong to omit groups from my analysis. Is there another type of ROC curve that isnt TPR vs FPR?