1
$\begingroup$

This question already has an answer here:

I am working on a fraud detection algorithm using a banking dataset which has large number of transactions. The number of true fraud cases are very small (<1%). So accuracy is not a good measure as if we say there is no fraud at all, we will still have over 99% accuracy. I learnt that AUC can be a good measure in such cases but I don't understand why. Can someone explain why?

$\endgroup$

marked as duplicate by Frans Rodenburg, mdewey, Michael Chernick, kjetil b halvorsen, mkt - Reinstate Monica Jul 15 at 8:32

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

migrated from stackoverflow.com Jul 2 at 8:35

This question came from our site for professional and enthusiast programmers.

6
$\begingroup$

Neither "classification" accuracy nor the $c$-index (concordance probability; AUROC) are proper accuracy scoring rules. "Accuracy" should be avoided at all costs, but the concordance probability is still a useful measure of pure predictive discrimination (separation of fraud and non-fraud on the basis of predicted probability of fraud). Concordance is the probability that of two chosen observations, one fraud and one non-fraud, the fraud is the one with a higher predicted probability. You can see how this will work fine even with extreme imbalance.

There are other measures to use that are more sensitive and statistically efficient. See for example http://fharrell.com/post/addvalue .

$\endgroup$
1
$\begingroup$

Accuracy is a legitimate validation metric when you are working with a balanced dataset. However, it is often the case, in classification problems, that there is a clearly majority-class. Also, errors are rarelly symetrical (for instance, in medicine, false positives and false negatives are not the same) For what I see in your question, you are already familiar with this concept.

On the other hand, AUC (I'd rather see the entire ROC curve, though) gives you an idea of how the true positive/true negative trade-off works. With this I mean that models with high AUC can detect a large amount of true positives without losing its ability to detect true negatives and vice-versa.

You may also be interested in the precision-recall curve

$\endgroup$
  • 2
    $\begingroup$ "Accuracy is a legitimate validation metric when you are working with a balanced dataset." The problem is, that there is another, completely different issue: Accuracy doesn't take costs into account. See many articles of Frank Harrell, or my answer here: stats.stackexchange.com/questions/368949/… . $\endgroup$ – Tamas Ferenci Jul 2 at 11:21
  • $\begingroup$ @TamasFerenci That's literally what is being stated on the very next sentence. "Also, errors are rarelly symetrical (for instance, in medicine, false positives and false negatives are not the same)" $\endgroup$ – David Jul 2 at 11:44
  • $\begingroup$ You're right, sorry, I didn't get "asymmetrical" at first glance. Nevertheless, I suggest not calling accuracy a "legitimate metric" in any case. Sorry again! $\endgroup$ – Tamas Ferenci Jul 2 at 11:46
  • $\begingroup$ @TamasFerenci Why not? If men and women write me messages at a similar rate and I want to predict whether the last anonymous message I received comes from a male or female, I cannot think of a single sitaution where getting the right answer 97% of the time is not enough of a reason to feel happy about my work! $\endgroup$ – David Jul 2 at 11:56
  • 3
    $\begingroup$ I can! If you really-really-really don't want to ever think of a sender of being female when he is indeed male (but you have much less problem with predicting a female sender as male), then you might very well prefer a model that never does the former error, and does the latter in 10% of the cases (accuracy: 90%), then one which never commits the latter, but commits the former in 3% of the cases (accuracy: 97%). $\endgroup$ – Tamas Ferenci Jul 2 at 12:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.