I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations.
http://blog.revolutionanalytics.com/2016/03/com_class_eval_metrics_r.html provides a great overview of different classification metrics.
Precision and Recall or
kappa both seem to be a good choice:
One way to justify the results of such classifiers is by comparing them to those of baseline classifiers and showing that they are indeed better than random chance predictions.
As far as I understand,
kappa could be the slightly better choice here, as random chance is taken into account. From Cohen's kappa in plain English I understand that
kappa deals with the concept of information gain:
[...] an Observed Accuracy of 80% is a lot less impressive with an Expected Accuracy of 75% versus an Expected Accuracy of 50% [...]
Therefore, my questions would be:
- Is it correct to assume
kappato be a better-suited classification metric for this problem?
- Does simply using
kappaprevent the negative effects of imbalance on the classification algorithm? Is re-(down/up)-sampling or cost-based learning (see http://www.icmc.usp.br/~mcmonard/public/laptec2002.pdf) still required?