I am working on a fraud detection algorithm using a banking dataset which has large number of transactions. The number of true fraud cases are very small (<1%). So accuracy is not a good measure as if we say there is no fraud at all, we will still have over 99% accuracy. I learnt that AUC can be a good measure in such cases but I don't understand why. Can someone explain why?
2
-
$\begingroup$ Check the following links. They have defined both concepts well. area-under-curve-of-roc-vs-overall-accuracy; Why-is-AUC-a-better-measure-of-an-algorithms-performance-than-accuracy; & advantages-of-auc-vs-standard-accuracy. $\endgroup$– rajat kabraCommented Jul 1, 2019 at 21:22
-
$\begingroup$ you can start here medium.com/usf-msds/… $\endgroup$– ImanCommented Jul 1, 2019 at 21:25
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Neither "classification" accuracy nor the $c$-index (concordance probability; AUROC) are proper accuracy scoring rules. "Accuracy" should be avoided at all costs, but the concordance probability is still a useful measure of pure predictive discrimination (separation of fraud and non-fraud on the basis of predicted probability of fraud). Concordance is the probability that of two chosen observations, one fraud and one non-fraud, the fraud is the one with a higher predicted probability. You can see how this will work fine even with extreme imbalance.
There are other measures to use that are more sensitive and statistically efficient. See for example http://fharrell.com/post/addvalue .
answered Jul 2, 2019 at 11:04
$\begingroup$
$\endgroup$
14
Accuracy is a legitimate validation metric when you are working with a balanced dataset. However, it is often the case, in classification problems, that there is a clearly majority-class. Also, errors are rarelly symetrical (for instance, in medicine, false positives and false negatives are not the same) For what I see in your question, you are already familiar with this concept.
On the other hand, AUC (I'd rather see the entire ROC curve, though) gives you an idea of how the true positive/true negative trade-off works. With this I mean that models with high AUC can detect a large amount of true positives without losing its ability to detect true negatives and vice-versa.
You may also be interested in the precision-recall curve
-
2$\begingroup$ "Accuracy is a legitimate validation metric when you are working with a balanced dataset." The problem is, that there is another, completely different issue: Accuracy doesn't take costs into account. See many articles of Frank Harrell, or my answer here: stats.stackexchange.com/questions/368949/… . $\endgroup$ Commented Jul 2, 2019 at 11:21
-
$\begingroup$ @TamasFerenci That's literally what is being stated on the very next sentence. "Also, errors are rarelly symetrical (for instance, in medicine, false positives and false negatives are not the same)" $\endgroup$– DavidCommented Jul 2, 2019 at 11:44
-
$\begingroup$ You're right, sorry, I didn't get "asymmetrical" at first glance. Nevertheless, I suggest not calling accuracy a "legitimate metric" in any case. Sorry again! $\endgroup$ Commented Jul 2, 2019 at 11:46
-
$\begingroup$ @TamasFerenci Why not? If men and women write me messages at a similar rate and I want to predict whether the last anonymous message I received comes from a male or female, I cannot think of a single sitaution where getting the right answer 97% of the time is not enough of a reason to feel happy about my work! $\endgroup$– DavidCommented Jul 2, 2019 at 11:56
-
3$\begingroup$ I can! If you really-really-really don't want to ever think of a sender of being female when he is indeed male (but you have much less problem with predicting a female sender as male), then you might very well prefer a model that never does the former error, and does the latter in 10% of the cases (accuracy: 90%), then one which never commits the latter, but commits the former in 3% of the cases (accuracy: 97%). $\endgroup$ Commented Jul 2, 2019 at 12:21