1
$\begingroup$

I have two classes and, in total, 40 subjects (20,20). I used an SVM to implement classification and did a leave-one-out cross validation to get an overall performance of the classifier.

How could I draw an ROC curve for this situation?

Is it even necessary to draw a ROC curve in this situation?

$\endgroup$
4
$\begingroup$

With only 40 subjects, the error rate will be highly discretised and probably not a very good indication of performance. The area under the ROC curve would be a better performance measure as it will be less discretised as it gives an indication of the accuracy of the ranking of the subjects (although of course it doesn't indicate how well the threshold is set). There is a really good tutorial on ROC analysis by Fawcett that should be helpful.

$\endgroup$
  • $\begingroup$ +1 for the Fawcett paper--it's pretty helpful. On the other hand, I don't think quantization of the error rate is that bad (100/40=steps of 2.5%). Area under the curve is definitely better, but depending on what you're doing and who you're talking to, I think the error rate is often useful too. $\endgroup$ – Matt Krause Apr 26 '12 at 9:51
  • $\begingroup$ cheers Matt, 2.5% seems pretty large to me; the key point is the variance of the classifier might be very high with only 40 subjects, so if you repeated the experiment with another 40 subjects then the error rate may be very different, but the difference in AUROCs are likely to be might smaller. Essentially the "error bars" on the performance statistics will need to be carefully considered as it is such a small sample. The error rate is certainly useful, I would present both statistics. $\endgroup$ – Dikran Marsupial Apr 26 '12 at 10:44
  • $\begingroup$ Thank you for both of your detailed reply. I'm a new hand. I was not very clear about "the error rate will be highly discretised". Do you mean this would happen in ROC curve? $\endgroup$ – herewish Apr 26 '12 at 13:32
  • 1
    $\begingroup$ Since you only have 40 examples, the error rate has to move in 100/40% steps (all correct: 100%, one wrong: 97.5%, two wrong: 95%, etc). I think @DikranMarsupial is suggesting that the AUROC will be less sensitive to this than the variance of the error rate. Seems reasonable to me, since you could get many possible shapes with 40 control points. $\endgroup$ – Matt Krause Apr 26 '12 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.