# ROC curve for two-sided cut-off

I am very, very confused about ROC curves. I have a Bayesian model which outputs a prevalence on a continuous scale between 0 and 1. I have a classification I would like to use that classifies that prevalence into an endemicity class based on a two sided cut-off (i.e. the prevalence cut off for being in the class is $0.35 <= P <= 0.6$.

So, I have the true observed prevalence, the predicted prevalence from my model (which is taken as the mean of the posterior distribution for that observation) I also have a probability for each prediction, that it belongs in the endemicity class of interest. This probability is an output of my Bayesian model and is simply the number of times the predicted prevalence did fall between those endemicity cut-offs divided by the number of posterior draws from my model (so a probability of $1$ means that all posterior realisations fell between 0.35 and 0.6).

Here is an example of my data:

> head(mydat)
Obs  Pred  pClass Class
1 0.441 0.503 0.56125     1
2 0.664 0.698 0.22225     0
3 0.252 0.468 0.58725     0
4 0.226 0.374 0.39325     0
5 0.014 0.107 0.02975     0
6 0.713 0.571 0.46425     0


Where:
Obs = the observed true prevalence
Pred = the predicted prevalence
Pclass = the proportion of posterior realisations from my model which fell within the endemicity class of interest.
Class - is the true class label.

I used the following R code to calculate what I think a ROC curve should be:

require(ROCR)
pred <- with( mydat , prediction( pred=Pclass ,labels=Class )
plot(perf <- performance(pred,'tpr','fpr') )


Producing this plot:

Have I calculated this correctly?
Have I used the right data?! Do I have the right data for calculating a ROC curve?
Can you produce a valid ROC curve for a classifier which has a two-sided cut-off?

I did read many tutorials on ROC curves, but my intuition fails me and I can't quite work out if I am applying it to my data in a valid way. Here is a copy/paste to put my data into R...

mydat <- structure(list(Obs = c(0.441, 0.664, 0.252, 0.226, 0.014, 0.713,
0.543, 0.777, 0.472, 0.512, 0.436, 0.312, 0.403, 0.709, 0.472,
0.625, 0.056, 0.335, 0.596, 0.679, 0.143, 0.68, 0.489, 0.319,
0.706, 0.789, 0.14, 0.261, 0.592, 0.05, 0.736), PRed = c(0.503,
0.698, 0.468, 0.374, 0.107, 0.571, 0.619, 0.693, 0.569, 0.584,
0.463, 0.03, 0.363, 0.562, 0.471, 0.313, 0.44, 0.448, 0.506,
0.617, 0.378, 0.643, 0.603, 0.403, 0.61, 0.743, 0.29, 0.666,
0.685, 0.044, 0.625), pClass = c(0.56125, 0.22225, 0.58725, 0.39325,
0.02975, 0.46425, 0.359, 0.2315, 0.4135, 0.419, 0.587, 0, 0.47025,
0.38625, 0.48825, 0.32425, 0.4725, 0.5615, 0.43975, 0.3605, 0.40825,
0.313, 0.411, 0.48525, 0.35475, 0.13475, 0.2725, 0.26875, 0.245,
5e-04, 0.316), Class = c(1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0,
1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0)), .Names = c("Obs",
"Pred", "pClass", "Class"), row.names = c(NA, -31L), class = "data.frame")


• Ok, thanks for the answer. So the variable pClass is actually a measure of the confidence that the observation belongs to the positive class. It is the proportion of times the model predicted a value within the range of the positive class (a value of 0.5 would mean that half the posterior realisations were between 0.35 and 0.6 whilst the other half were either below or above this threshold). It sounds like I can use this to do exactly as you suggest (which is what I have done)? Commented Jun 16, 2015 at 16:11
• Yes, in that case you can use pClass directly. Commented Jun 16, 2015 at 16:15