Are you sure you need a distinct ROC curve per patient? What exactly are you going to do with the AUC measures of which you have one per patient?

If you want to asses your classifier's performance, you could also randomly group your datapoints into 10 folds. The patient number would be an additional column in your data-set which your classifier can use. When you repeat that cross validation 3 times, you have 30 samples of the classifiers performance.

If the goal is to classify datapoints from unknown future patients (starting from their first datapoint), you should disregard patient numbers alltogether.

Edit: You can always use macro averages of you ROC measurements or of F-measures. Usually, you would take each of both classes once as positive and once as negative and average the two performance scores. You could also give more weight to the rare class and compute a weighted macro average. This is quite unusual, but so is the approach of having this many performance measures which are all based on one patient. (Forman 2010)

Are you sure you need a distinct ROC curve per patient? What exactly are you going to do with the AUC measures of which you have one per patient?

If you want to asses your classifier's performance, you could also randomly group your datapoints into 10 folds. The patient number would be an additional column in your data-set which your classifier can use. When you repeat that cross validation 3 times, you have 30 samples of the classifiers performance.

If the goal is to classify datapoints from unknown future patients (starting from their first datapoint), you should disregard patient numbers alltogether.

Edit: One option is Cohen's Kappa. It takes care of the no information rate. It is defined when there are no TP. Is is however undefined when the classification is perfect which may create another problem with small test-sets.

If you should know the concrete misclassification costs of FP versus FN (and perhaps the profits of TP and TN), then you should always use those as your performance metric.

You can always use macro averages of you ROC measurements or of F-measures. Usually, you would take each of both classes once as positive and once as negative and average the two performance scores. You could also give more weight to the rare class and compute a weighted macro average. This is quite unusual, but so is the approach of having this many performance measures which are all based on one patient. (Forman 2010)

Are you sure you need a distinct ROC curve per patient? What exactly are you going to do with the AUC measures of which you have one per patient?

If you want to asses your classifier's performance, you could also randomly group your datapoints into 10 folds. The patient number would be an additional column in your data-set which your classifier can use. When you repeat that cross validation 3 times, you have 30 samples of the classifiers performance.

If the goal is to classify datapoints from unknown future patients (starting from their first datapoint), you should disregard patient numbers alltogether.

Are you sure you need a distinct ROC curve per patient? What exactly are you going to do with the AUC measures of which you have one per patient?

If you want to asses your classifier's performance, you could also randomly group your datapoints into 10 folds. The patient number would be an additional column in your data-set which your classifier can use. When you repeat that cross validation 3 times, you have 30 samples of the classifiers performance.

If the goal is to classify datapoints from unknown future patients (starting from their first datapoint), you should disregard patient numbers alltogether.

Edit: You can always use macro averages of you ROC measurements or of F-measures. Usually, you would take each of both classes once as positive and once as negative and average the two performance scores. You could also give more weight to the rare class and compute a weighted macro average. This is quite unusual, but so is the approach of having this many performance measures which are all based on one patient. (Forman 2010)