ROC graph shape

Could you explain to me how the shape of a ROC curve is determined?

From the following illustration, it seems that for every time the actual class (C) is positive, it goes up and when it's negative, it goes to the right.

Why does it only seem to be based on the true class rather than the actual right or wrong classification of the object? I assume it's to do with the mentioned score, but I don't know how this would be calculated and ranked. Please explain.

Thank you.

-
Reading the question is not easy with that animated image :) – Kasper Aug 12 '14 at 14:10
@Kasper Do you mean it's distracting or is my question not clear with that illustration? – Glenn Aug 12 '14 at 14:12
It is just distracting, but I understand it is useful to understand the topic. – Kasper Aug 12 '14 at 14:13
@Kasper I understand. I just thought that was the clearest illustration I could give of my problem/question. Hope it doesn't keep you from answering (if you can). – Glenn Aug 12 '14 at 14:15

In the table, the subjects have been sorted by their score. This score is intended to characterize the subjects as positive (high score in this example) or negative (low score here).

As the true positives are on the y-axis, each positive lets the graph go up, as you observed. As the false positives are on the x-axis, each negative (only those can be false positive) lets the graph go right. Now you see how the "disorder" in the ordered table (ordered by the score, but its absolute value is of no convern any more) shapes the curve: If PNPNPNP..., the estimated ROC curve would look like stairs. This is associated with the worst possible classification: Although you have sorted the subjects by their score, the N and P are still perfectly mixed. So the score doesn't give a clue about the true status. Flipping a coin is as bad.

In turn, PPPPP...NNNNN would be perfect classification: All P are above some certain score and all N are below this score, the perfect cutoff. The ROC curve is then the upper left border of the plot, reaching the point of 100% true positives and 0% false positives.

In reality, the ROC curve is somewhere between these extreme cases. There you have to trade false positives against false negatives, as already noted by Kasper ad Alexis.

(Note by the way that choice of the cutoff due to estimated ROC curve is too optimistic concerning the probabilities of true classification. This is because we would choose a upper left vertex of the plot and this point is typically more to the upper left than the true unknown ROC curve would ever be.)

-
Thank you Horst. That makes a lot of sense. So if I understand correctly the subjects that have been classified have been given a score (high for positive, low for negative). And then ranked. If these would all have been classified correctly, they would all be grouped together (PPP..NNNN). If not, it couses a mix (and stairs shape). The only thing I'm still unsure of is how this score is calculated for these subjects? – Glenn Aug 12 '14 at 16:02
The score is something measured by our scientific client. E.g. it could be the measured blood glucose level if you want to diagnose diabetes. The ROC curve says nothing about it. All it says is how many and where in the ranking misclassified subjects are. Personally, I find it didactically misleading that the score in your example is between 0 and 1. – Horst Grünbusch Aug 12 '14 at 16:28
I see. I use a linear SVM. In this context, does that mean the 'Score' column represents the distance to the hyperplane (transformed to fit the 0 to 1 form)? For example: a subject gets a positive score in the vector dimensional space (and classified on the positive side of the hyperplane) even though it is negative. When ranked by score, this subject gets mixed between the real positive ones and create the stairs shape. Is that correct? – Glenn Aug 12 '14 at 17:58
@Glenn: Yes. But you don't even have to transform to $[0,1]$ for this to happen. – Horst Grünbusch Aug 13 '14 at 9:08
Wonderful, thank you so much! – Glenn Aug 13 '14 at 17:22

The ROC curve is useful to determine a good compromise for the cut of level of a classification problem. In a naive way of looking, you set the cut of level so you have the lowest classification error. In your example, you could set the cut of level of the score (the third column) at 0.5. If above, it is classified as P, if below, classified as N

But in real life it might be that a classifying a negative as positive has a far higher cost as the inverse. Take the example of the diagnosis of cancer, you don't really want to tell somebody he is ok, where he actually has cancer (false positive). The other error, telling him he has cancer where he actually doesn't, isn't that bad as the patient will probably just be retested.

The ROC curve teaches us that we have to make a compromise between false positive rate and true positive rate. If for example we would not allow a false positive rate higher than 1%, we have to live with live with a pretty high true negative rate.

In the graph, the false positive rate is on the x-axis, where the true positive rate is on the y-axis.

That is more or less how I remember it, but it can very well be that I switched the terms around, as I always have to puzzle it out again. Pretty sure the wikipedia page http://en.wikipedia.org/wiki/Receiver_operating_characteristic will tell you everything about it.

Hope this helped as a first introduction.

-
I disagree with Kaspar that false positives are necessarily worse than false negatives. I teach my epidemiology students that answering the question of the relative importance of sensitivity vs. specificity depends on the consequences of being Type I wrong vs. Type II. Cancer and HIV are excellent examples of where sensitivity is more important. However, tests informing immediate intervention with irreversible consequences require high specificity. Confirmation of which limb to operate on/amputate is an example. – Alexis Aug 12 '14 at 14:57
Thank you for your answers Kasper and @Alexis . I still don't quite get why the graph seems to be generated according to the actual class (pos =up, neg=right), instead of generating is based on whether the classification of that instance was right or wrong. – Glenn Aug 12 '14 at 15:46

I will use your animated image to illustrate the calculation. So, the table from the right side contains 20 data instances. These instances are indexed by their rank when ordered by the value from the third column, called score. Now, what is that score? Let's suppose that we have two classes/labels, named $1$ and $0$. That score is a numerical value which resembles the probability that that instance to be of class $1$. Because ROC curves deals only with binary case (I know there are extensions, but it does not matter here), than the probability that the same instance to be classified as the label $0$ is $1-Score$. The second column tells us which class the specific instance actually is considered.

Usually, if the scores resembles probabilities (as I considered, and if it is not the case, one can easily transform scores to range $[0,1]$ to look like probabilities), a classifier predict an instance as class $1$ if the score for class $1$ is greater than the score for class $0$. If the sum of the scores is $1$, as I supposed, than is easy to see that this condition is the same as: predict an instance to be of class $1$ if score of that instance is greater than $0.5$. The main idea is that it uses a threshold value to discriminate in order to predict.

Now that we clarified what that information is we develop the idea behind ROC curves. ROC curves tells us how the classifier would behave for all useful values of the threshold, not only for the arbitrary value $0.5$.

It starts from the threshold value of $1$ and drops incrementally until $0$. But which are the useful values for the threshold? For the threshold value of $1$ would give $TPR=0$ since no value from Score column is greater than $1$, and similarly $FPR=0$. One can note also that the threshold value of $0.95$ gives the same amounts. More, all the values of the threshold between $1$ and $0.9$ give the same values. The conclusion is that the only useful values for the running threshold are the distinct values from score column. So the ROC curve points will be computed only for those distinct values of score.

One step further is to consider the threshold value of $0.9$. The $TPR =0.1$ because from all the $P=10$ instances which are labels actually as positive, only $TP=1$ has the score greater than the threshold values and is predicted as positive. You can follow wikipedia page and compute the same value for $FPR$.

After doing that, you will obtain the coordinates of a point which will be part of the ROC curve. Then you peek the next increasing value of the threshold from all the useful values and repeat the algorithm. Finally you will end up in the right corner where all the actual $1$ instances are predicted as $1$ (TPR=1) and all the actual instances $0$ will be also predicted as $1$ and (FPR=1), the final point and the end of the ROC curve.

-
Thank you. Could you also tell me how these scores (resembling probabilities) are calculated? – Glenn Aug 12 '14 at 16:40
The score does not represent any probability. You can see this because this 0.505 in the table cannot be calculated from these 20 observations. The score is some measurement, e.g. a blood parameter. Then medics check if this blood parameter is above of below some threshold in order to get the diagnosis. Now this blood parameter would be useless as diagnostic indicator if too many healthy patients have their blood parameter in a apparently sick region. That's what the ROC curve tells us. And not the score itself. – Horst Grünbusch Aug 12 '14 at 16:40
I never said that the scores ARE probabilities, I only said that the scores can be made to looks like probabilities in order to justify the common threshold of 0.5. I see the purpose of ROC curves to tell us how well-separated are the positive and negative cases, when used score as a measure. I often use scores in range [0,1] because they are often use in many bayesian models (there score are probabilities), some discriminant models (there the scores are not always probabilities but looks like), and talking in terms of probabilties seems more insightful to me at least. – rapaio Aug 12 '14 at 16:57
@Glenn the scores which resembles probabilities is the job of the classifier algorithm. In a decision tree the score is the proportion in a leaf node of positive cases over the count of all cases. And even if we are in the case where we have a measure called Score and we want to see how that measure separates $0$ from $1$, it still can be considered as a classification problem, where the classifier is obviously made from the single condition if(threshold_value>Score)than $1$ else $0$. – rapaio Aug 12 '14 at 17:04
@rapaio I see. I use a linear SVM. In this context, does that mean the 'Score' column represents the distance to the hyperplane (transformed to fit the 0 to 1 form)? For example: a subject gets a positive score in the vector dimensional space (and classified on the positive side of the hyperplane) even though it is negative. When ranked by score, this subject gets mixed between the real positive ones and create the stairs shape. Is that correct? – Glenn Aug 12 '14 at 17:23