56
$\begingroup$

I'm having trouble understanding the ROC curve.

Is there any advantage / improvement in area under the ROC curve if I build different models from each unique subset of the training set and use it to produce a probability? For example, if $y$ has values of $\{a, a, a, a, b, b, b, b\}$, and I build model $A$ by using $a$ from 1st-4th values of $y$ and 8th-9th values of $y$ and build model $B$ by using remained train data. Finally, generate probability. Any thoughts / comments will be much appreciated.

Here is r code for better explanation for my question:

Y    = factor(0,0,0,0,1,1,1,1)
X    = matirx(rnorm(16,8,2))
ind  = c(1,4,8,9)
ind2 = -ind

mod_A    = rpart(Y[ind]~X[ind,])
mod_B    = rpart(Y[-ind]~X[-ind,])
mod_full = rpart(Y~X)

pred = numeric(8)
pred_combine[ind]  = predict(mod_A,type='prob')
pred_combine[-ind] = predict(mod_B,type='prob')
pred_full          = predict(mod_full, type='prob')

So my question is, area under ROC curve of pred_combine vs pred_full.

$\endgroup$
  • 3
    $\begingroup$ A better example would do a lot to improve the question. $\endgroup$ – mpiktas Jul 2 '14 at 8:17
  • $\begingroup$ My understanding is that you want to increase AUC by choosing some specific samples? If that is your purpose, I strongly believe that this approach of biased sample selection is completely wrong, at least if your purpose is to find a good measure for classification performance. $\endgroup$ – rapaio Jul 2 '14 at 8:19
  • 2
    $\begingroup$ have a look at interactive demo of ROC $\endgroup$ – Alleo Dec 2 '15 at 0:09
170
$\begingroup$

I'm not sure I got the question, but since the title asks for explaining ROC curves, I'll try.

ROC Curves are used to see how well your classifier can separate positive and negative examples and to identify the best threshold for separating them.

To be able to use the ROC curve, your classifier has to be ranking - that is, it should be able to rank examples such that the ones with higher rank are more likely to be positive. For example, Logistic Regression outputs probabilities, which is a score you can use for ranking.

Drawing ROC curve

Given a data set and a ranking classifier:

  • order the test examples by the score from the highest to the lowest
  • start in $(0, 0)$
  • for each example $x$ in the sorted order
    • if $x$ is positive, move $1/\text{pos}$ up
    • if $x$ is negative, move $1/\text{neg}$ right

where $\text{pos}$ and $\text{neg}$ are the fractions of positive and negative examples respectively.

This nice gif-animated picture should illustrate this process clearer

building the curve

On this graph, the $y$-axis is true positive rate, and the $x$-axis is false positive rate. Note the diagonal line - this is the baseline, that can be obtained with a random classifier. The further our ROC curve is above the line, the better.

Area Under ROC

area under roc

The area under the ROC Curve (shaded) naturally shows how far the curve from the base line. For the baseline it's 0.5, and for the perfect classifier it's 1.

You can read more about AUC ROC in this question: What does AUC stand for and what is it?

Selecting the Best Threshold

I'll outline briefly the process of selecting the best threshold, and more details can be found in the reference.

To select the best threshold you see each point of your ROC curve as a separate classifier. This mini-classifiers uses the score the point got as a boundary between + and - (i.e. it classifies as + all points above the current one)

Depending on the pos/neg fraction in our data set - parallel to the baseline in case of 50%/50% - you build ISO Accuracy Lines and take the one with the best accuracy.

Here's a picture that illustrates that and for details I again invite you to the reference

selecting best threshold

Reference

$\endgroup$
  • $\begingroup$ Thanks for the great comment! I am sorry that I couldnt upvote for your answer it seems I need to have more than 15 reputation :( $\endgroup$ – Tay Shin Jul 2 '14 at 19:48
  • $\begingroup$ Just curious, your step size would have to depend on the number of positive/negative labels produced by your classifier correct? I.e. In the gif, the step size upwards is .1, if you had an extra positive label (in place of a negative label), then the "curve" would end up at 1.1 on the vertical axis (or maybe I am missing something?). So, in that case your step size needs to be smaller? $\endgroup$ – kyphos Mar 8 '15 at 15:50
  • $\begingroup$ No, the step size depends on the number of positive/negative examples in the dataset, not on the accuracy of the classifier. $\endgroup$ – Alexey Grigorev Mar 9 '15 at 6:32
  • 2
    $\begingroup$ @gung understood. Alexey : instead of positive and negative examples, I think it should be: true positives and false positives. You may be able to see my edition of the answer, which was reverted by gung. thanks $\endgroup$ – Escachator Jun 15 '16 at 16:05
  • 3
    $\begingroup$ @AlexeyGrigorev, love the reply you give and vote up. I am not sure if there are two ROC definitions. I am referring to the ROC definition here (en.wikipedia.org/wiki/Receiver_operating_characteristic), the x-axis should be false positive rate, which is (# of predictions to be positive, but should be negative) / (# of total negative), I think in the reference, I think the x-axis is not drawing false positive rate, which does not consider the (# of total negative)? $\endgroup$ – Lin Ma Aug 28 '16 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.