9
$\begingroup$

In my data, I have a class variable, denoted as $C$. This class variable values are ${0, 1}$ (binary). Almost all observations of $C$ are 0 (close to 100%, more precisely, 97%). I would like a "performance" test on different classification models (it could be accuracy). What I am afraid of happening is that if I have a classification model that always classifies any observation into class 0, then that model will be 97% accurate (even though it never considers any other variables).

Are there any well known performance tests for classification models on data dealing with very rare events?

Improve this question
$\endgroup$

5 Answers 5

Reset to default
3
$\begingroup$

A few possibilities come to my mind.

Looking at the overall hit rate is usually not a very good idea as it will depend on the composition of the test set if the performance for the different classes differs. So at the very least, you should be specify (and justify) the relative frequency of the classes in your test data in order to derive a meaningful value.

Secondly, as @Shorack already said, specify which types of error are how important. Often, the classifier needs to meet certain performance criteria in order to be useful (and overall accuracy is rarely the adequate measure). There are measures like sensitivity, specificity, positive and negative precdictive value that take into account the different classes and different types of misclassification. You can say that these measures answer different questions about the classifier:

  • sensitivity: What fraction of cases truely belonging to class C is recognized as such?
  • specificity: What fraction of cases truely not belonging to class C is recognized as such?
  • positive predictive value: Given the classifier predicts class C, what is the probability that this prediction is correct?
  • negative predictive value: Given the classifier predicts that the case is not form class C, what is the probability that this prediction is correct?

These questions often allow to formulate specifications that the classifier must need in order to be useful.

The predictive values are often more important from the point of view of the practical application of the classifier: they are conditioned on the prediction, which is the situation you are in when applying the classifer (a patient usually is not interested in knowing how likely the test is to recognize diseased cases, but rather how likely the stated diagnosis is correct). However, in order to properly calculate them you need to know the relative frequencies of the different classes in the population the classifier is used for (seems you have this information - so there's nothing that keeps you from looking at that).

You can also look at the information gain that a positive or negative prediction gives you. This is measured by positive and negative likelihood ratio , LR⁺ and LR⁻. Briefly, they tell you how much the prediction changes the odds towards the class in question. (see my answer here for a more detailed explanation)

For your trivial classifier, things look like this: I'll use the "0" class as the class in question, so "positive" means class "0". Out of 100 cases, 100 are predicted positive (to belong to class 0). 97 of them really do, 3 don't. The sensitivity for class 0 is 100% (all 97 cases truely belonging to class 0 were recognized), specificity is 0 (none of the other cases were recognized). positive predicitve value (assuming the 97:3 relative frequency is representative) is 97%, negative predictive value cannot be calculated as no negative prediction occurred.

$LR^+ = \frac{\text{sensitivity}}{1 - \text{specificity}} = 1$
$LR^- = \frac{1 - \text{sensitivity}}{\text{specificity}} = \frac{0}{0}$
Now LR⁺ and LR⁻ are factors with which you multiply the odds for the case to belong to the positive class ("0"). Having an LR⁺ of 1 means that the positive prediction did not give you any information: it will not change the odds. So here you have a measure that clearly expresses the fact that your trivial classifier does not add any information.

Completely different direction of thoughts: You mention that you'd like to evaluate different classifiers. That sounds a bit like classifier comparison or selection. The caveat with the measures I discuss above is that they are subject to very high random uncertainty (meaning you need lots of test cases) if you evaluate them on "hard" class labels. If your prediction is primarily continuous (metric, e.g. posterior probability) you can use related measures that look at the same kind of question but do not use fractions of cases but continuous measures, see here. These will also be better suited to detect small differences in the predictions.

(@FrankHarrell will tell you that you need "proper scoring rules", so that is another search term to keep in mind.)

Improve this answer
$\endgroup$
3
$\begingroup$

First of all: are all hits equally important and all misses equally important? If so, then there is nothing wrong with your null-model scoring that good: it simply is an excellent solution.

If you find it important to have a good performance on predicting the 1's, you could use the F-measure instead. It is basically the harmonic mean of recall (what portion of the actual 1's have been predicted as 1) and precision (what portion of the predicted 1's were actually a 1). For a model to score high on this measure, it needs to:

  1. Find most of the 1's.
  2. Not often predict a 1 when it is actually 0.

And it needs to do both simultaneously. Even if your model does only one of the 2 in almost a perfect manner, it will have a low score if it does not perform on the other requirement. https://en.wikipedia.org/wiki/F1_score

Improve this answer
$\endgroup$
1
  • $\begingroup$ That is an improper scoring rule that uses only 1 bit of information from the predictions. Improper scoring rules are optimized by bogus models. $\endgroup$
    – Frank Harrell
    Commented Aug 1, 2013 at 11:21
2
$\begingroup$

I'm glad that @cbeleites opened the door ... The concordance probability or $c$-index, which happens to equal the ROC area in the special case of binary $Y$, is a nice summary of predictive discrimination. The ROC curve itself has a high ink:information ratio, but the area under the curve, because it equals the concordance probability, has many nice features, one of them being that it is independent of the prevalence of $Y=1$ since it conditions on $Y$. It is not quite proper (use generalized $R^2$ measures or likelihood ratio $\chi^2$ to achieve that) and is not sensitive enough to be used to compare two models, it is a nice summary of a single model.

Improve this answer
$\endgroup$
1
$\begingroup$

The Receiver Operating Characteristic (ROC) http://en.wikipedia.org/wiki/Receiver_operating_characteristic curve and associated calculations ( namely Area Under Curve- AUC) are commonly used. basically you imagine your classifier gives a continuous reponse ( eg between 0 and 1) and you plot the sensitivity vs false alarm rate (1- specificity) as the decision threshold varies between 0 and 1. These were specifically designed for rare events ( spotting enemy planes?).

Improve this answer
$\endgroup$
1
$\begingroup$

When you are dealing with strongly imbalanced data, the Precision-Recall curve is a very good tool, better than its more common cousin the ROC curve.

Davis et. al. have shown that an algorithm which optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.