I have a classifier that I am using to classify patient data into one of two classes.
Specifically, I have many patients, and for each patient I have a number of (multivariate) datapoints, recorded once daily. For each daily datapoint, the classifier takes that datapoint and classifies it into one of two classes. The two classes are highly imbalanced, perhaps around 90%-vs-10%.
I initially wanted to compute a ROC curve for each patient, and then look at the spread of AUC across all patients, in order to get an idea of how reproducible my classifier is. However, occasionally, a patient has no positive cases, i.e. all the datapoints belong to the majority class (the negative class). In this case it is nonsensical to talk about ROC curve.
I could of course discard patients with no positives, but that would bias my average AUC measure. That would also be the case if I simply said AUC=0.5 (no information) for patients with no positives.
Are there any sensible and unbiased alternatives to ROC curves that I could use to gauge the quality of the classifier on this patient population?
(For clarity: I don't care about the timeseries dynamics here, I am simply classifying each datapoint in isolation)