3
$\begingroup$

A ROC point is a point with a pair of x and y values in the ROC space where x is 1 – specificity and y is sensitivity and the ROC curve always starts at (0.0, 0.0) and ends at (1.0, 1.0).

What about a "perfect" classifier that makes at some threshhold no mistakes, i.e. sensitivity and specificity are both 1 at this threshhold. For this threshhold the ROC point is (0.0, 1.0). Using the start of the ROC curve (0.0, 0.0) for that classifier would mean that there are two different y values (0 and 1) for the same x value (0).

I know it will be uncommon to find a classifier that makes perfect classification at some threshhold but having this question makes me wonder whether I understand the ROC curves.

$\endgroup$
  • $\begingroup$ In fact, the number of y values along the y-axis is equal to the number of positive instances in the data set (possible thresholds). $\endgroup$ – lnathan May 7 at 14:11
4
$\begingroup$

The problem arises because of the assumption that there can not be different $y$ values for one $x$ value. You seem to (implicitly?) mix up the ROC curve with a graph that depicts a function of the form $y = f(x) = x + ...$ where indeed each $x$ value can be connected to a single $y$ value. This is not the case for a ROC curve. In fact the website you linked says further below that A classifier with the perfect performance level shows a combination of two straight lines – from the origin (0.0, 0.0) to the top left corner (0.0, 1.0) and further to the top right corner (1.0, 1.0).

Let's consider an example. Here is some data where test values can seperate perfectly the occurence of some entity in the reality with the perfect treshold in red:

data

The ROC curve (red line) for this data looks as the description in the quote before suggests:

ROC curve

As you can see, for $x= 0$ there is not only $y= 0$ and $y= 1$ but in fact we actually get for every treshold with $x= 0$ a point in the interval $[0, 1]$ on the $y$ axis, hence there can be many more different $y$ values for $x= 0$ than the two that you mentioned (start (0.0, 0.0) and end (1.0, 1.0)). The answer is therefore, that a ROC curve always starts at (0, 0), even in case of a "perfect classifier". And I hope that it has become clear that this is not a contradiction.

You can reproduce this with the following R code:

# data
# sample size
n <- 20
# test values
test <- 1:n
# actual diagnosis
reality <- factor(rep(c(FALSE, TRUE), each= n/2), levels= c(TRUE, FALSE))
# vector of tresholds
treshold <- c(-Inf, seq(min(test), max(test), 0.5), Inf)

# calculate sensitivity and specifiticity per treshold
results <- sapply(treshold, function(treshold_i){ 
  # what cases are over the treshold?
  test_results <- factor(test  >= treshold_i, levels=c(TRUE, FALSE))
  # create a table
  table_results <- table(test_results, reality)
  # estimate sensitivity
  sens <- table_results[1, 1]/ sum(table_results[ , 1])
  # estimate specifiticity
  spec <- table_results[2, 2]/ sum(table_results[ , 2])
  # save both and the used treshold in a matrix
  m <- matrix(c(treshold_i, sens, spec), ncol= 3)
  # return matrix
  return(m)
})

# organize the data
# flip matrix
results <- t(results)
# name columns
colnames(results) <- c("treshold", "sens", "spec")

# ROC curve (2nd plot)
plot(1 - results[ , "spec"], results[ , "sens"], type= "l", col= "red",
     xlab= "1 - Specificity", ylab= "Sensitivity")
# diagonal line
abline(0, 1)

# first plot: test values vs reality
plot(test, as.logical(reality), ylab= "reality")
# line for treshold with just correct classification
# in the roc curve this is the point (0, 1)
abline(v= 10.5, col= "red")
# table with the results
results
$\endgroup$
  • 1
    $\begingroup$ Thank you! Now I got it: the mapping from a treshold value t to a (FPR,TPR) is unique as pointed out by Calimo. But as you describe, this does not mean that x can only be mapped to one y in a ROC curve (and that was my wrong assumption). $\endgroup$ – ErKanns May 8 at 9:20
  • 1
    $\begingroup$ A great first answer, welcome to the site. $\endgroup$ – Zhubarb May 8 at 10:56
3
$\begingroup$

A ROC does not summarize a classifier, it summarizes a range of possible classifications for a prediction. A classifier is summarized by just one point in on the axis of TPR / FPR.

For instance, even if you had a great biomarker for breast cancer, the biomarker expression levels are summarized with a ROC. You will still have to decide at what concentration of mRNA you will say, "this assay is from a cancer" vs "this assay is from a benign/noncancerous growth".

With a perfect predictor, it's still possible to choose an expression range or value that says, "No, this isn't cancer" for any possible finding. It's a dumb kind of prediction, but possible in every scenario. In that case, you're never wrong when it's not cancer, but you're always wrong when it is. So it "starts" at 0,0. It just so happens that as you consider intermediate values of threshold, the TPR increases while the FPR remains at 0 until you reach the maximum level of the controls (perfect 0,1 prediction). Then at even higher threshold, you start declaring False Negatives and the curve tends to 1,1 in an upside down "L" shape until you declare all assays cancerous.

$\endgroup$
2
$\begingroup$

If you have a perfect classifier, since it will make no mistakes, your FPR and TPR are $0$ and $1$ respectively, so, you're right on (0,1) point in the ROC plane. However, this isn't a curve; classifiers are represented as points in ROC.

The question should be restated for a perfect predictor as @AdamO pointed out, in which we really have the curve because now we have a set of classifiers, which represent a set of points in ROC plane, therefore a curve, going from (0,0) to (1,1) as it should be. We still start from the origin because perfect prediction is not perfect classification. It's all about the choice of the tuning parameter. Adam's example in his 3rd paragraph is the execution of this idea.

Before editing this answer, I was actually thinking of a model $M$ with some tuning parameter (e.g. like threshold) having no effect on the result because it was meant to be a perfect classifier and try to visualize what happens when we change the tuning parameter. In that case, it starts from (0,1), but it actually stays there and doesn't move towards (1,1), simply because the parameter has no effect.

$\endgroup$
  • 5
    $\begingroup$ No, the ROC curve always starts from (0, 0). Also what do you mean by "some sort of a parameter"? $\endgroup$ – Calimo May 7 at 18:34
  • $\begingroup$ @Calimo I was also thinking about the bias term in binary SVM, which can be used for ROC generation. And, thanks, it was first your comment that I've realized something is terribly wrong. $\endgroup$ – gunes May 7 at 19:43
2
$\begingroup$

It helps if you think at which threshold $t$ the curve hits a given point. For all ROC curves (built from dataset $D$ where higher data values are indicative of positive results, and assuming $D \notin \pm\infty$) you have the following points that you mentioned:

  • $t = +\infty \rightarrow (0, 0) $
  • $t = -\infty \rightarrow (1, 1) $

These decision thresholds $t$ will classify everything as negative, respectively positive. They aren't very useful in everyday life, but they are the keys that binds every ROC curve to (0, 0) and (1, 1)

The threshold for your "perfect" prediction will never be an infinity. So the threshold $\min D < c_{opt} < \max D$ yielding a perfect classification will fall somewhere in between:

  • $t = c_{opt} \rightarrow (0, 1) $

In this case the mapping from a value $t$ to a $(FPR, TPR)$ is unique.

$\endgroup$
  • $\begingroup$ You say In this case the mapping from a value x to a (FPR,TPR) is unique. This seems to contradict to the answer from boulder because in that answer one gets different y (TPR) values for the same x (FPR) value. But the arguments from boulder seem to be correct to me which makes me confused. $\endgroup$ – ErKanns May 8 at 8:39
  • $\begingroup$ Ok the confusion here is that I am using x to refer to your data, and not the x in the plot. Let me rephrase it with an other letter like d for data. $\endgroup$ – Calimo May 8 at 8:56
  • $\begingroup$ I am confused. t is the treshold if I get it right. But why does it say "where higher data values d are indicative of positive results" not "where higher data values t are indicative of positive results"? Also D is the set of all thresholds t? If so, why does it say that D∉±∞ if later we talk about (-) ∞ thresholds t? $\endgroup$ – ErKanns May 8 at 9:11
  • $\begingroup$ @ErKanns No, D is your data, and t is the decision threshold you use to classify this data as positive or negative. $\endgroup$ – Calimo May 8 at 9:17
  • $\begingroup$ I am sorry, what is d? The sentence "where higher data values d are indicative of positive results" seems to me to imply that you talk about the treshold but you call the treshold later on t. $\endgroup$ – ErKanns May 8 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.