How can I tell if my binary classifier is any good?

Question

Say I have a data set with 10,000 rows and the target is a binary variable with 1500 positives (1's) and 8500 negatives (0's). I run a model and get predictions on the 0-1 interval. My question is what's the best way to tell if my model is actually adding any value?

If the AUC is ~ to (1 - the positive % of the target variable) (in this case an AUC of ~.85) does this mean the model is not really adding any value? Or do I need to look at other measures (ie Sensitivity, Specificity, False Pos, False Neg, etc.)?

Any suggestions would be appreciated.

Answer 1

In some sense, I think it depends on what you mean by "adding value," but let's assume you are simply trying to judge whether your particular model is doing better than guessing 0 for all your cases or even randomly guessing.

Since you are talking about AUC, I'm assuming you are drawing an ROC curve based on the cutoff for a logistic classifier. If you have any sort of area under the curve (well, greater than 0.5), then you are doing better than random. The confusion, I think, lies in comparing an AUC of 0.85 with the overall accuracy of 0.85 for the "always-guessing-zero" case — this is not what AUC measures. This case is, in fact, a single point on the ROC diagonal (point (0,0), since you never guess positive).

However, the key lies in the "adding value" comment. Since you have an AUC $ \approx $ 0.85, there is likely some choice of threshold which provides you with a "reasonable" classifier. Here, the end goal of your classifier has to be considered. Which errors are more important for you to minimize? There's always a trade-off. Here is where you'd want to consider things like precision & recall, the F1 score, etc.

A useful reference I've found which discusses ROC analysis in more detail is: Fawcett, T (2006). An introduction to ROC analysis. Pattern Recognition Letters 27(8): 861–874.

Answer 2

Another standard approach is to randomly split your data into training and validation datasets. Fit your model to the training data and then evaluate how well the predicted values fit the validation dataset using ROC.

The Wikipedia article on ROC (http://en.wikipedia.org/wiki/Receiver_operating_characteristic) discusses the ROC perhaps not being the best statistic for comparing models. Alternatives to the ROC would be computing the Brier score or deviance.

Answer 3

Here's another way of expressing "added value": the likelihood ratios ($LR^+$ and $LR^-$).

If your model predicts "positive", $LR^+$ tells you how much the odds increase, that the case you predict is truely positive.
In other words, $\text{odds after testing} = \text{odds before testing} \cdot LR^+$. Likewise, if the test predicts negative, use $LR^-$ to multiply the pre-test odds with ($LR^-$ is < 1, if the test is any good).

Here's how they are calculated forom sensitivity and specificity:

$LR^+ = \frac{\text{sensitivity}}{1 - \text{specificity}}$; $LR^- = \frac{1 - \text{sensitivity}}{\text{specificity}}$

The great thing about these likelihood ratios is that they quantify the amount of information towards "positive" the test gives you and at the same time they are independent of the prevalence (relative frequency) of positives in the tested population.

Of course, if you know the prevalence of positives in your test population, you may directly go on and calculate positive and negative predictive values

---

In any case, if you interpret the ROC make sure you are talking about a working point, i.e. a (sensitivity; specificity)-pair, that actually makes sense for your application.

Personally, I hardly ever think about AUC and focus instead of parts of the curve that are sensible for the application.