Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 6 down vote favorite
2

I am evaluating and combining a few binary classification models. I am using the ROC and PR curves to evaluate their performance. The problem I am having is that as I try to improve the method, I am improving the AUC-ROC but the PR curve suffers. For example: ROC curve PR curve

As an aside, I am actually adding a weak learner to Method 1 to arrive at Method 2, and then adding another weak learner or two to arrive at Method 3. When I was only evaluating AUC-ROC, it looked fine, but when I saw the PR curve, it seems I have been degrading the performance. Now it seems that the weak learners are doing better at points lower in the ranked list. But this is only for one dataset training/test split. What would be a principled way to investigate what is going on and come up with a way to use the weak learners so as to improve both ROC and PR curves?

Update:

To visualize this, I am showing the weak learner that I am adding to Model 1 to arrive at Model 2 here: weak ROC weak PR

share|improve this question
Probably this is specific to machine learning, but isn't doing AUC-ROC enough? What extra information does a PR curve provide? – suncoolsu Sep 3 '11 at 0:39
@suncoolsu, well, they have the same information, since you can reproduce one from the other, but PR curves usually are more informative visually if you are interested in predicting positive class examples and when the number of negative class points is way larger. This is usually the case in information retrieval, for example. – highBandWidth Sep 6 '11 at 15:26
@highBandWith 1. From the PR-curve I derive a heavy class skew. From the jump of the red curve in the last plot I suspect, that Recall=0.01 does not correspond to 1000 points absolute. So: Since I further assume that the curves represent some average across multiple splits, could be so kind to plot the error bars, too ? – steffen Nov 15 '11 at 15:10
@highBandWith 2. I think that in a case of such a heavy class skew one has a hard time to optimize both curves simultaneously. What is the goal ? Are you allowed to missclassify an arbitrary number of negative instances ? Or it is more important to identify some positives with high precision, so that recall is low ? Or is this a more a theoretical question (which nevertheless would be very interesting !) ? – steffen Nov 15 '11 at 15:16
1  
First make sure you need to combine different prediction systems vs. choosing a single more comprehensive method (e.g., penalized regression with many predictors in a single model; random forests). If you really need to combine multiple risk predictors to get a better risk prediction, search Bayesian model averaging (BMA) or bepress.com/ucbbiostat/paper222. One form of BMA weights competing models by the probability that each model is the "correct" one (if such a thing exists) using Bayes factors (see Adrian Raftery's work). – Frank Harrell Nov 19 '11 at 3:52
show 4 more comments

4 Answers

up vote 1 down vote

I will state a few things about the ROC / PR spaces that are surely evident for you but that I prefer to make clear.

  • The ROC space is on the $x$-axis one minus the specificity : $1-Sp$, and on the $y$-axis the sensitivity : $Se$.

  • The PR space is on the $x$-axis the recall, which is an other name of the sensitivity : $Re = Se$, and on the $y$-axis the precision, which is an other name of the Positive Predictive Value : $Pr = PPV$ ;

  • If $p$ is the probability of being in the "positive class", we have $$Pr = PPV = {Se\cdot p \over (1-Sp)\cdot(1-p) + Se \cdot p}.$$

The "horizontal slices" in the ROC space correspond to "vertical slices" of PR space. From the above equality, it is easy to see that when in the ROC space a curve (eg the red curve of your first graph) is on the left of a second one (the green curve), in the PR space the corresponding (red) curve is above the (green) curve.

This is the case in your second graph, except for Recall values $< 0.1$. The corresponding part of the ROC curves in your first graph is for Se $< 0.1$ which is "glued" to the $y$-axis, and you can’t see anything. Here the advantage of the PR space is that it helps visualizing this area.

So I don’t see contradiction in these results : method 3 is indeed better than the two others, except for Sensitivity / Recall values $< 0.1$, which correspond to very high Specificity values.

The morality is that the way you improve your classifier slightly degrades its performances when you demand it to have a very high Specificity.

These are quite trivial reflexions, but who knows, this can help?

share|improve this answer
1  
Nice addition. Why do other fields keep inventing new names (e.g., precision-recall) for old concepts? – Frank Harrell Jan 5 '12 at 16:16
@FrankHarrell This is even worse: if you interpret your classification procedure as a test (which seems legitimate to me), you have $Sp = 1 - \alpha$ and $Se = 1 - \beta$. I think one of the reasons is that these notions are created by people in totally different fields and later connected to stats... – Elvis Jan 5 '12 at 21:32
up vote 0 down vote

For imbalanced classes using AUC as a measure of classifier performance, rather than (0,1)-loss can be misleading. See for example Xue and Titterington "Do unbalanced data have a negative effect on LDA?". For two-class classification the (0,1)-loss is usually the loss of real interest, so you may find that working directly with that loss, rather than AUC, is more informative.

share|improve this answer
How would using (0,1)-loss help in combining the scores? – highBandWidth Nov 18 '11 at 20:32
I'm not sure that it would. That is, I don't know your exact setting so I don't know what the most appropriate loss function for you would be (but (0,1) is probably a reasonable bet). What I do know is that AUC can be uninformative in the case of imbalanced classes (as your own PR plots demonstrate). For example when the number of negative examples is much greater than the number of positive examples then a large numerical change in the number of false positives might mean only a small change in the false positive rate in the ROC analysis. (Continued... – Bob Durrant Nov 18 '11 at 22:44
...More) So what I'm proposing is that if the performance measure you've adopted doesn't let you measure the performance, then you need to consider an alternative. – Bob Durrant Nov 18 '11 at 22:47
up vote 0 down vote

I eventually resorted to using logistic regression (and similar models like adaptive splines) etc. to combine the scores. I think the idea is that of stacking and has been used before, e.g., here and here.

share|improve this answer
up vote 0 down vote

Deviance (or -2 log likelihood) is the most statistically sensitive measure. I would use that to compare models.

share|improve this answer
I agree that the log likelihood is the correct measure. Do you evaluate that on the training set or the test set? I would assume the test set, otherwise we'd be overfitting, unless we can compute the full model likelihood (after integrating out the parameters) as in a bayesian framework. I don't think my algorithm can integrate out the parameters from the model. – highBandWidth Jan 5 '12 at 17:05
To use split-sample validation requires over 20,000 subjects in many cases. So before answering what is your total sample size and the fraction of observations used in the training set? – Frank Harrell Jan 5 '12 at 23:21
I have way more then 20,000 points, though there is a large class imbalance (~.1% positive). Wouldn't the likelihood (without integrating out parameters) over the training set be vulnerable to over-fitting? – highBandWidth Jan 5 '12 at 23:38
1  
You might need 7000 positives before data splitting is a competitive method of model validation. I wasn't trying to imply that you would just use the likelihood in the training sample (although that is better than using the likelihood from a very small test sample) but rather using resampling, e.g., 100 repeats of 10-fold cross-validation or 300 bootstrap repetitions. And regarding likelihood you can convert it to one of the unitless indexes (logarithmic scoring rule or generalized $R^2$). – Frank Harrell Jan 6 '12 at 14:12
ohh got it! That helps a lot. If you don't mind you or someone else (I?) could add this to the answer. I love this site because I can learn from previous questions and answers. – highBandWidth Jan 7 '12 at 21:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.