3
$\begingroup$

I'm very new to machine learning (this is my second project and only the first significant one) and here's the problem I'm currently dealing with:

I am evaluating the performance of a model that outputs a percentage as a final result (which is not meant to be thresholded to create separate categories). This result is compared against the estimation given by pathologists, and I've been asked to create a ROC curve testing different parameters for my model.

The issue is, as far as I understand it, that can only be done for a binary 0/1 output. I suppose I could set a margin within which my model would be judged to be 'accurate' but I don't see how I could divide my results between true/false positives/negatives.

It seems to me that this is simply not an applicable approach for my situation, but I might be mistaken....Is there a way to produce a ROC curve in such a situation?

Improve this question
$\endgroup$
4
  • $\begingroup$ What does "the estimation given by pathologists" look like? Is that binary? $\endgroup$
    – Calimo
    Commented Oct 25, 2016 at 12:15
  • $\begingroup$ @Calimo, the pathologists' estimation is also a percentage $\endgroup$
    – user136151
    Commented Oct 25, 2016 at 12:41
  • $\begingroup$ So I'm not sure why you'd want to apply binary classification tools. What makes you think a ROC curve would be appropriate? Why not calculate the correlation? $\endgroup$
    – Calimo
    Commented Oct 25, 2016 at 13:17
  • 2
    $\begingroup$ I don't think the ROC is appropriate or even applicable but I've been asked to do it regardless... As there is no way to dichotomise my data and create a system that would provide me with TP/TP and TN/FN categories, I think I'll just have to discuss this with my supervisor and tell her it's not the most appropriate tool to evaluate this model. $\endgroup$
    – user136151
    Commented Oct 25, 2016 at 13:41

1 Answer 1

Reset to default
2
$\begingroup$

Not a simple way, but you can create multiple ROC curves and average them (Landgrebe and Duin 2007). There are different ways in which to do the averaging as explained by (Forman and Scholz 2009) in the context of CV, but the same concepts apply.

Improve this answer
$\endgroup$
2
  • $\begingroup$ if I understood these papers properly, they both deal with data that is already categorised in true/false positives/negatives/ groups and models that divide the data in a finite number of categories. This is not my case as my final output value is continuous and will not be thresholded or split into a discrete number of groups in any way. $\endgroup$
    – user136151
    Commented Oct 25, 2016 at 13:52
  • $\begingroup$ The second one yes, the first one no. But the same reasoning applies. As the first paper states (or recollects from other papers), you can generalize ROC to multiple classes by using one vs. all or one vs. one classification. (This means that you are going to train multiple classifiers. If your classifier is capable of multi class per default, you might want to avoid this by using Fmeasure instead of ROC.) Then you have multiple ROC curves that you need to average or merge. The second paper discusses this in another context, granted. But the same reasoning can be applied to your case. $\endgroup$
    – David Ernst
    Commented Oct 25, 2016 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.