I'm studying machine learning and find an example question on the book which really confused me.


A scoring classifier is evaluated on a test set of 10 examples resulting in the following probability scores:

           0.9, 0.8, 0.7, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1

with true classes:

           1     1    0    1    1    0    1    1    0    0

The answer for the ROC curve is Could anyone give me some hints on how they draw this? I'm very confused on why the curve turn right when the tpr raise to approx 0.3 and 0.7. Thanks!

  • $\begingroup$ please add the self-study tag $\endgroup$ – Antoine Jan 17 '17 at 8:17
  • 1
    $\begingroup$ @Antoine done! ty $\endgroup$ – newinjava Jan 17 '17 at 8:19

Create dataset of actual tag and score. Sort dataset by decreasing score. For every row, do the following (treat as rough pseudocode):

# calculate these
total.class.1.seen = # number of class-1 cases >= row.score
seen.cases = # number of cases seen till now.
total.class.1 = # total class 1 cases in data
total.class.0 = # total class 0 cases in data.

TPR = total.class.1.seen/total.class.1
FPR = (seen.cases - total.class.1.seen)/total.class.0

Now plot these TPR and FPR values. Essentially what you're doing at every row, is saying that anything which has score >= this score would be classified as class-1.

The above code doesn't do well when you have non-unique scores (As you have it for 0.7 here). Think of how you want to handle such cases. *Hint: for all unique scores, the plot would be a sequence of only vertical or horizontal line-segments, when non-unique, you get slant line segments.

| cite | improve this answer | |
  • $\begingroup$ What language is this? $\endgroup$ – Nick Cox Jan 17 '17 at 9:32
  • $\begingroup$ It isn't any language, It is supposed to work as rough pseudo-code. Probably my usage of formatting is wrong? if so, suggestions are welcome. $\endgroup$ – Ujjwal Kumar Jan 17 '17 at 9:36
  • $\begingroup$ I didn't try to read it as I knew it wasn't any language I used! I've no objection to formatting pseudocode; I just suggest that it is labelled at the top as such. $\endgroup$ – Nick Cox Jan 17 '17 at 9:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.