3
$\begingroup$

I am very, very confused about ROC curves. I have a Bayesian model which outputs a prevalence on a continuous scale between 0 and 1. I have a classification I would like to use that classifies that prevalence into an endemicity class based on a two sided cut-off (i.e. the prevalence cut off for being in the class is $ 0.35 <= P <= 0.6 $.

So, I have the true observed prevalence, the predicted prevalence from my model (which is taken as the mean of the posterior distribution for that observation) I also have a probability for each prediction, that it belongs in the endemicity class of interest. This probability is an output of my Bayesian model and is simply the number of times the predicted prevalence did fall between those endemicity cut-offs divided by the number of posterior draws from my model (so a probability of $1$ means that all posterior realisations fell between 0.35 and 0.6).

Here is an example of my data:

> head(mydat)
    Obs  Pred  pClass Class
1 0.441 0.503 0.56125     1
2 0.664 0.698 0.22225     0
3 0.252 0.468 0.58725     0
4 0.226 0.374 0.39325     0
5 0.014 0.107 0.02975     0
6 0.713 0.571 0.46425     0

Where:
Obs = the observed true prevalence
Pred = the predicted prevalence
Pclass = the proportion of posterior realisations from my model which fell within the endemicity class of interest.
Class - is the true class label.

I used the following R code to calculate what I think a ROC curve should be:

require(ROCR)
pred <- with( mydat , prediction( pred=Pclass ,labels=Class )
plot(perf <- performance(pred,'tpr','fpr') )

Producing this plot: enter image description here

Have I calculated this correctly?
Have I used the right data?! Do I have the right data for calculating a ROC curve?
Can you produce a valid ROC curve for a classifier which has a two-sided cut-off?

I did read many tutorials on ROC curves, but my intuition fails me and I can't quite work out if I am applying it to my data in a valid way. Here is a copy/paste to put my data into R...

mydat <- structure(list(Obs = c(0.441, 0.664, 0.252, 0.226, 0.014, 0.713, 
0.543, 0.777, 0.472, 0.512, 0.436, 0.312, 0.403, 0.709, 0.472, 
0.625, 0.056, 0.335, 0.596, 0.679, 0.143, 0.68, 0.489, 0.319, 
0.706, 0.789, 0.14, 0.261, 0.592, 0.05, 0.736), PRed = c(0.503, 
0.698, 0.468, 0.374, 0.107, 0.571, 0.619, 0.693, 0.569, 0.584, 
0.463, 0.03, 0.363, 0.562, 0.471, 0.313, 0.44, 0.448, 0.506, 
0.617, 0.378, 0.643, 0.603, 0.403, 0.61, 0.743, 0.29, 0.666, 
0.685, 0.044, 0.625), pClass = c(0.56125, 0.22225, 0.58725, 0.39325, 
0.02975, 0.46425, 0.359, 0.2315, 0.4135, 0.419, 0.587, 0, 0.47025, 
0.38625, 0.48825, 0.32425, 0.4725, 0.5615, 0.43975, 0.3605, 0.40825, 
0.313, 0.411, 0.48525, 0.35475, 0.13475, 0.2725, 0.26875, 0.245, 
5e-04, 0.316), Class = c(1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 
1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0)), .Names = c("Obs", 
"Pred", "pClass", "Class"), row.names = c(NA, -31L), class = "data.frame")
$\endgroup$
1
$\begingroup$

Can you produce a valid ROC curve for a classifier which has a two-sided cut-off?

No, not directly. To compute an ROC curve you have to be able to sort examples based on the confidence you have that they belong to the positive class. Sorting based on the score your models output doesn't do that.

You could attempt to shoe-horn your predictions into such a level of confidence, for instance by using distance to the center of your positive interval as a criterion, but usually such hacks are a bad idea ...

$\endgroup$
  • $\begingroup$ Ok, thanks for the answer. So the variable pClass is actually a measure of the confidence that the observation belongs to the positive class. It is the proportion of times the model predicted a value within the range of the positive class (a value of 0.5 would mean that half the posterior realisations were between 0.35 and 0.6 whilst the other half were either below or above this threshold). It sounds like I can use this to do exactly as you suggest (which is what I have done)? $\endgroup$ – Simon O'Hanlon Jun 16 '15 at 16:11
  • $\begingroup$ Yes, in that case you can use pClass directly. $\endgroup$ – Marc Claesen Jun 16 '15 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.