6
$\begingroup$

I am using ROC curves and full AUC values to compare different models, using simulated data. Now I think I am confused with the interpretations of ROC curves and AUC values. Please see the figure below (sorry it is partial from screen shots...)

There are three models compared, and I know that the model shown in green should preform best of all. However, as you can see, the green curve is superior to the other two before the FPR reaching around 0.2. This cut-off of 0.2 is quite interesting: it is the percentage of differentially expressed genes that I specify in my simulation (i.e. 20% of the observations are simulated to be positives).

My concern are:

  1. given that people in reality will seldom choose a FPR cut-off of 0.5 or higher, why people would prefer a ROC curve with FPR ranging from 0 to 1 and use the full AUC value (i.e. calculate the entire area under the ROC curve) instead of just reporting the area made from, say, 0 to 0.25 or to 0.5? Is that called "partial AUC"?

  2. in the figure below, what can we say about the performances of the three models? The AUC values are: green (0.805), red (0.815), blue (0.768). The red curve turns out to be superior, but as you see, the superiority is only reflected after FPR > 0.2. Thanks :)

enter image description here

$\endgroup$
  • 2
    $\begingroup$ This is also a great example of how ensembling many similar models (AUC ranging from .76-.82) can increase accuracy. In this case, a good meta-model based on the green and red models would be better than either of them on their own. $\endgroup$ – Zach Nov 1 '13 at 14:07
5
$\begingroup$

I agree with your concerns.

given that people in reality will seldom choose a FPR cut-off of 0.5 or higher, why people would prefer a ROC curve with FPR ranging from 0 to 1 and use the full AUC value (i.e. calculate the entire area under the ROC curve) instead of just reporting the area made from, say, 0 to 0.25 or to 0.5? Is that called "partial AUC"?

  • I'm a big fan of having the complete ROC, as it gives much more information that just the sensitivity/specificity pair of one working point of a classifier.
  • For the same reason, I'm not a big fan of summarizing all that information even further into one single number. But if you have to do so, I agree that it is better to restrict the calculations to parts of the ROC that are relevant for the application.

in the figure below, what can we say about the performances of the three models? The AUC values are: green (0.805), red (0.815), blue (0.768). The red curve turns out to be superior, but as you see, the superiority is only reflected after FPR > 0.2. Thanks :)

  • That depends entirely on your application. In your example, if high specificity is needed, then the green classifier would be best. If high sensitivity is needed, go for the red one.

As to the comparison of classifiers: there are lots of questions and answers here discussing this. Summary:

  • classifier comparison is far more difficult than one would expect at first
  • not all classifier performance measures are good for this task. Read @FrankHarrells answers, and go for so-called proper scoring rules (e.g. Brier's score/mean squared error).
$\endgroup$
4
$\begingroup$

Usually, it will be your application that will determine whether your focus is on precision or recall.

@2 These will be dramatically different in the medical field, where you will often tolerate having a bad precision for the sake of a very good recall, when it comes to prevention, i.e. you prefer to label a lot of healthy people as sick and make additional tests, rather than to let someone die (here sickness is considered "relevant", and labeled as sick "retrieved").

On the other hand, in production, you can tolerate a certain quota of bad apples and you might prefer a test that does not catch all the faulty products but is much more precise in identifying the bad apples - usually the costs associated with inspecting the items can not be disregarded. This corresponds to a high precision, and low recall scenario.

For your models, either you know what you need and pick a better model for that purpose, or you pick the one with better AUC. Of course there are also other things you might take into consideration, such as, which model is more parsimonious (has fewer explanatory variables), where are the assumptions better met, etc.

@1 I don't see the advantage of putting less information in a plot, especially if it could be misleading. (unless you work in marketing)

$\endgroup$
  • $\begingroup$ thanks a lot! I think in my field we are more interested in FPR less than 0.2. Few people would detect differentially expressed genes with a cut-off of p-value greater than 0.2. So is ROC curve the correct one to use or the PR curve? $\endgroup$ – alittleboy Oct 30 '13 at 21:06
  • 1
    $\begingroup$ Pick the model that suits your purpose, considering the relative costs of false positives and false negatives. AUC can be a misleading measure of the quality of a model, particularly in cases like this where you don't care about all the possible predictor-variable values (in this case, you apparently don't care about values that give FPR > 0.2). It's good that you plotted the curves; they indicate that for your purpose at low FPR the green model has superior sensitivity, as you thought. $\endgroup$ – EdM Oct 30 '13 at 21:50
1
$\begingroup$

You didn't state the ultimate goal of the exercise, hence the choice of ROC curves was not well motivated. Many useful things can be done with log-likelihood and Brier scores, as well as with the distribution of predicted risks (ignoring $Y$). The use of cutoffs is questionable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.