1
$\begingroup$

Consider a binary classification problem. Here I do data partitioning 100 times randomly into training (90%) and test sets (10%). I get the True positive rate (TPR) and False positive rates (FPR) for the 100 iterations. So effectively I get 100 ROC plots. Is there any way to combine the 100 plots to generate a single plot. Plotting these 100 into one makes it very clumsy. Is it alright to take all FPRs and TPRs into two big vectors and then take unique elements in it and then plot a single graph. But when I do this i generate a graph something like the one below.

And for another case of partitioning, I get the ROC graph as below.

As per my understanding, ROC curve is an increasing one and does not have valleys in the graph. If we see the graphs above there are valleys in the plot. Is there any other way to combine all the 100 ROC plots that i get for the 100 different partitionings?

$\endgroup$
  • 1
    $\begingroup$ Rather than trying to combine TPR and FPR, which are fractions, I would think you should combine the underlying confusion matrices (i.e. counts), before computing the fractions. $\endgroup$ – GeoMatt22 May 9 '17 at 21:19
3
$\begingroup$

Fawcett in An introduction to ROC analysis, section 8 describes two algorithms to properly average multiple ROC curves into one: Vertical (8.1) and Threshold (8.2) averaging.

These two algorithms allow the construction of a proper, ie. monotone, and smooth ROC curve, with error bars.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

What you do is basically cross-validation – with the difference that examples may be part of training or test-set multiple times because you select them randomly.

Rather than partitioning the data 100 times I would (randomly) partition them into 100 equally size folds just once. If you use every fold just once as the test set, then you get the same 100 times 90% vs 10% iterations, but you get only a single response for every example, because every example is only once part of a test set.

You then can use the responses of all examples to compute a single ROC curve for your 100 iterations just the same way you would do for a single run on a bigger test set.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.