2
$\begingroup$

I have a classifier that I am using to classify patient data into one of two classes.

Specifically, I have many patients, and for each patient I have a number of (multivariate) datapoints, recorded once daily. For each daily datapoint, the classifier takes that datapoint and classifies it into one of two classes. The two classes are highly imbalanced, perhaps around 90%-vs-10%.

I initially wanted to compute a ROC curve for each patient, and then look at the spread of AUC across all patients, in order to get an idea of how reproducible my classifier is. However, occasionally, a patient has no positive cases, i.e. all the datapoints belong to the majority class (the negative class). In this case it is nonsensical to talk about ROC curve.

I could of course discard patients with no positives, but that would bias my average AUC measure. That would also be the case if I simply said AUC=0.5 (no information) for patients with no positives.

Are there any sensible and unbiased alternatives to ROC curves that I could use to gauge the quality of the classifier on this patient population?

(For clarity: I don't care about the timeseries dynamics here, I am simply classifying each datapoint in isolation)

Improve this question
$\endgroup$
6
  • $\begingroup$ You cannot compute a ROC curve for each patient. The points that shape the (empirical) ROC curve are due to different patients. You question is still vague. Should your classifier use the whole time series or just one single measurement, and you just ask when to measure it best? $\endgroup$
    – Horst Grünbusch
    Commented Oct 17, 2016 at 13:26
  • $\begingroup$ Do you have both the true and predicted values for each patient for each day? $\endgroup$
    – mdewey
    Commented Oct 17, 2016 at 14:44
  • $\begingroup$ Please clarify what you mean when you say: "For each daily datapoint, I classify the datapoint into one of two classes." Is the classification done independently of the multiple covariates you record, or have you already built some model that classifies based on the covariates? $\endgroup$
    – EdM
    Commented Oct 17, 2016 at 14:49
  • $\begingroup$ @mdewey: Yes, not sure how I'd calculate ROC curve otherwise $\endgroup$
    – funklute
    Commented Oct 17, 2016 at 14:56
  • $\begingroup$ @HorstGrünbusch: I'm classifying one datapoint at a time, and the classifier takes only that (multivariate) datapoint as input (i.e. no timeseries input, and implicitly using iid assumption). I'm predicting multiple datapoints for each patient, so I clearly can have a ROC curve for each patient...did I misunderstand something there? $\endgroup$
    – funklute
    Commented Oct 17, 2016 at 15:05

1 Answer 1

Reset to default
1
$\begingroup$

Are you sure you need a distinct ROC curve per patient? What exactly are you going to do with the AUC measures of which you have one per patient?

If you want to asses your classifier's performance, you could also randomly group your datapoints into 10 folds. The patient number would be an additional column in your data-set which your classifier can use. When you repeat that cross validation 3 times, you have 30 samples of the classifiers performance.

If the goal is to classify datapoints from unknown future patients (starting from their first datapoint), you should disregard patient numbers alltogether.

Edit: One option is Cohen's Kappa. It takes care of the no information rate. It is defined when there are no TP. Is is however undefined when the classification is perfect which may create another problem with small test-sets.

If you should know the concrete misclassification costs of FP versus FN (and perhaps the profits of TP and TN), then you should always use those as your performance metric.

You can always use macro averages of you ROC measurements or of F-measures. Usually, you would take each of both classes once as positive and once as negative and average the two performance scores. You could also give more weight to the rare class and compute a weighted macro average. This is quite unusual, but so is the approach of having this many performance measures which are all based on one patient. (Forman 2010)

Improve this answer
$\endgroup$
5
  • $\begingroup$ There is a rather large variation in performance from patient to patient, so the idea is to measure this variation. While overall I want the best classifier, I also want to minimise patient-to-patient variation, thus a ROC curve per patient. I'm not sure I understood the two next paragraphs...are you talking about using patient number as an additional predictor? (there is absolutely no information in the patient number unfortunately) $\endgroup$
    – funklute
    Commented Oct 17, 2016 at 13:14
  • $\begingroup$ When you say there is a large performance variation from patient to patient. Does this mean classification performance or the measurements themselves? If the patient's measurements vary between patients, but not much within patients, then there is per definition information in the patient number. If you should use it depends on how this system should be deployed in the future. $\endgroup$
    – David Ernst
    Commented Oct 17, 2016 at 13:33
  • $\begingroup$ @user7019377: ok, I see what you're saying, and yes there is information in that sense. But I still don't see how that relates to testing performance, given a previously trained model? $\endgroup$
    – funklute
    Commented Oct 17, 2016 at 15:01
  • $\begingroup$ Well your initial question is hard to answer since those performance metrics who can deal with class and cost imbalance have weaknesses regarding small test sets and accuracy is not adequate when you have cost imbalance in addition to simple class imbalance. At the same time, I am not convinced that your specific experimental setup is necessary. More information about the application of this classifier could help to find an experimental setup that doesn't have those complications. $\endgroup$
    – David Ernst
    Commented Oct 17, 2016 at 15:05
  • $\begingroup$ point well taken, I think the application is a little too hairy to go into detail here, and indeed part of the point is to have a starting point from which various experimental setups can be gauged... I may well need to go back to the drawing board and come up with a more tightly defined question, but will have a look at the above links you posted, thanks! $\endgroup$
    – funklute
    Commented Oct 17, 2016 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.