2
$\begingroup$

I have a binary classification set with about 900 samples. I first use kfold cross validation and AUC of the ROC curve to determine which classifiers perform best. Once I've done that, I use a leave-one-group out approach for additional metrics. In my dataset, the data is collected among 32 groups. Groups within my set are uneven. It can happen for example that one group will have 32 class0 samples, and 0 class1 samples. One result of classification could then be

Class 0 [30 2]
Class 1 [0  0]

However, in this case, tpr = TP/(TP+FN) = 0 / (0+0) and I get a NaN. My question is what should I do in this case? My plan was to use AUC to compare classifiers using leave-one-group-out approach, but this will obviously not work because of what I've just mentioned. Ive thought of ignoring groups like this, but there are more than 1 and it feels wrong to omit groups from my analysis. Is there another type of ROC curve that isnt TPR vs FPR?

Thanks

$\endgroup$
  • 1
    $\begingroup$ It really depends what you want to do with the number, which isn't sufficiently clear. I don't think it's unreasonable to ignore the groups since you do indeed have no information about TPR (sensitivity) in these cases. $\endgroup$ – Ben Bolker May 6 '17 at 23:23
  • $\begingroup$ I'm using AUC as a metric to determine which classifier works best. By ignoring a group, I run in to the possibility of choosing a classifier which works best on few groups of data, potentially leading to an overfit model $\endgroup$ – jerpint May 8 '17 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.