Alternative to ROC curve when some test cases have no real positives? I have a classifier that I am using to classify patient data into one of two classes.
Specifically, I have many patients, and for each patient I have a number of (multivariate) datapoints, recorded once daily. For each daily datapoint, the classifier takes that datapoint and classifies it into one of two classes. The two classes are highly imbalanced, perhaps around 90%-vs-10%.
I initially wanted to compute a ROC curve for each patient, and then look at the spread of AUC across all patients, in order to get an idea of how reproducible my classifier is. However, occasionally, a patient has no positive cases, i.e. all the datapoints belong to the majority class (the negative class). In this case it is nonsensical to talk about ROC curve.
I could of course discard patients with no positives, but that would bias my average AUC measure. That would also be the case if I simply said AUC=0.5 (no information) for patients with no positives.
Are there any sensible and unbiased alternatives to ROC curves that I could use to gauge the quality of the classifier on this patient population? 
(For clarity: I don't care about the timeseries dynamics here, I am simply classifying each datapoint in isolation)
 A: Are you sure you need a distinct ROC curve per patient? What exactly are you going to do with the AUC measures of which you have one per patient?
If you want to asses your classifier's performance, you could also randomly group your datapoints into 10 folds. The patient number would be an additional column in your data-set which your classifier can use. When you repeat that cross validation 3 times, you have 30 samples of the classifiers performance.
If the goal is to classify datapoints from unknown future patients (starting from their first datapoint), you should disregard patient numbers alltogether.
Edit:
One option is Cohen's Kappa. It takes care of the no information rate. It is defined when there are no TP. Is is however undefined when the classification is perfect which may create another problem with small test-sets.
If you should know the concrete misclassification costs of FP versus FN (and perhaps the profits of TP and TN), then you should always use those as your performance metric.
You can always use macro averages of you ROC measurements or of F-measures. Usually, you would take each of both classes once as positive and once as negative and average the two performance scores. You could also give more weight to the rare class and compute a weighted macro average. This is quite unusual, but so is the approach of having this many performance measures which are all based on one patient.
(Forman 2010)
