2
$\begingroup$

I think I should rephrase my question after reading a few replies. The original question is kept intact at the bottom. So maybe I should ask the question this way: if you use a ROC curve to find the predictive power of your predictor, will the slope on the ROC curve be a good metric to rely on?

Assume there is a undiscovered relationship f(x) = x(1-x), where 0 <= x <= 1. We also assume f(x) is a good predictor for our class label y. Now all you have is some (x, y) pairs. If you want to see how x predicts y, you generate a ROC curve and get your AUC, you would get a plot similar to what I have shown below, and my question about the predictive power of x on y remains the same.

Hope this is clearer.

------------------- original question ------------------------------ Is it true that the slope of a ROC implies prediction performance, i.e., the bigger the slope a segment on the ROC curve is, the better the prediction the segment corresponds to.

Take a look at the following ROC curve

enter image description here

Can I assume the beginning part (TP rate 0~0.2) and the ending part (FP rate 0.85 ~ 1) predict much better than the middle segment?

The reason I am asking is that I am wondering if I can throw away data points in the middle and use only data points correspond to the segments at the two ends. Does this make sense?

Thank you!

p.s. I understand I could have reversed the predictor so that AUC > 0.5, but the questions remain the same.

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ The AUC curves relate to the classifier itself and not to some data points so I don't understand what throw away data points in the middle means. In some sense, I guess that the F-score is likely to be higher where the slope is higher but then your main focus is on what is the F-score of a certain point and choose your classifier threshold to be there. $\endgroup$
    – Gabizon
    Commented Oct 18, 2016 at 13:12
  • $\begingroup$ Hi, I'm not sure I understand how you are using ROC and for which purpose. Each point on the ROC-curve is a parametric function of the threshold $t$ i.e $\left(x(t),y(t)\right)=\left( FPR(t), TPR(t) \right)$ evaluated for all datapoints. You can see it as if you're plotting the confusion matrix of the classification at each threshold. This means you can't 'throw away' data points as each segment will correspond to all data points. Should I clarify? $\endgroup$
    – ragulpr
    Commented Oct 18, 2016 at 13:19
  • $\begingroup$ @Gabe The curve is actually just formed by connecting the dots (points) on the curve, and every dot (point) corresponds to a value of the predictor. For example, if you use height to predict if someone is a basketball player, and you have 10k people with height information. You use height as the predictor (classifier), then you have actually 10k points on the curve, although they are connected on the curve and look consecutive. If it turn out either short people or tall people are more likely to be basketball players, then we would see a ROC curve like the one I showed above. $\endgroup$
    – Mousheng Xu
    Commented Oct 18, 2016 at 13:20
  • $\begingroup$ @ragulpr I understand your point of view. However, I think you are assuming the predictive power of the predictor is monotonic in respect to its value, which may not be always true I am afraid. See my comments to Gabe. $\endgroup$
    – Mousheng Xu
    Commented Oct 18, 2016 at 13:22
  • $\begingroup$ Yes that's right but it's not the monotonicity that has me questioning. We only need a the argsort/ranking $r_i$ of the scores and some class label $c_i$ for each datapoint $i=1,\ldots,n$ to create a ROC. I'm confused about what the input data $r$ and $c$ is. Could you provide us with the code snippet you use to create the graph? $\endgroup$
    – ragulpr
    Commented Oct 18, 2016 at 13:29

3 Answers 3

Reset to default
2
$\begingroup$

Drawing the ROC curve leads to more confusion. Think about what is at the root of your issue. You can't interpret pieces of the ROC curve without having a utility/cost/loss function. The area under the ROC curve happens to equal the $c$-index (concordance probability) which is a simple interpretable pure measure of predictive discrimination.

Instead of saying something is a "good predictor of a class label" use "good predictor of the probability of class membership" and think hard about measuring the predictive discrimination in the model's predicted probabilities (especially if the model is well calibrated).

If you have correctly modeled the function form of $x$ (here by, for example, fitting a quadratic effect in a binary logistic model), and you get an AUROC ($c$-index) below 0.5, you have made a computational error.

Improve this answer
$\endgroup$
1
$\begingroup$

From your curve, I would recommend you to flip your predicted labels. In most cases, the ROC curve should be always above the $y=x$ line. Because random guess will get $y=x$.

Secondly, it is not possible to "throw away some data points", the curve is evaluated from all the data points in every threshold value.

In sum, I would suggest you to flip your predicted labels, and try to evaluate the performance again.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yeah, actually I put a "p.s." clause in my post. Thanks. $\endgroup$
    – Mousheng Xu
    Commented Oct 18, 2016 at 13:45
1
$\begingroup$

Your intuition about the segments of the ROC curve with high slope implying "good prediction" is incorrect.

When it comes to an ROC curve, the definition of a "positive" prediction is simply that this value makes it past your prediction threshold. (Remember, the points in an ROC curve are ordered based on your probability prediction.) The threshold is something that varies along the ROC curve. For example, the right edge of an ROC curve, at which all ROC curves have both a 1.0 TPR and 1.0 FPR, simply means that your threshold is zero (i.e. you're counting all cases in the data set as "positive").

In general, a region of high slope in ROC means that the "ones" in your data are represented in that region at a disproportionately high rate. Low slope regions means the "ones" are represented at a disproportionately low rate in that region.

The way to interpret your ROC curve is as follows. Most of the true "ones" in the data are in the highest and the lowest predicted probability region, while most of your "zeros" are in the middle portion. So if you trust the "high slope" segments, as you suggest, then you're trusting the value of your low probability predictions, even though the "ones" are largely represented there. That would be a mistake. What we typically want from our binary classifiers is for the "ones" to have a high probability prediction, and the "zeros" to have a low probability prediction.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.