1
$\begingroup$

I'm looking for a good metric to compare binary classification methods for a task where

  1. The data is highly imbalanced.
  2. The approximate data imbalance is unknown.

There are certainly more than 100 negative examples for every positive one. However, how much more is unknown. It may be 1:1000 or 1:100000 or more.

In this situation, precision doesn't seem to make sense to use as a metric, because we don't know what the real imbalance is and precision will change depending on that ratio. ROC values (true positive rate and false positive rate) have a real meaning regardless of the ratio. However, the AUROC is very close to 1 and the ROC curve approaches looking like a 90 degree corner. Comparing an AUROC of 0.999 and 0.9999 isn't very intuitive.

Is there a metric for such a situation that allows for an intuitive comparison between different models?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

Yes, there is the precision-recall gain curve which is for severely imbalanced data where the class distribution can vary between tasks and you want to compare performance between them. It is straightforward to implement and published in NIPS. It works by standardising precision to the baseline chance expectation. This is for where you care about the positive class, but not negative.

https://papers.nips.cc/paper/5867-precision-recall-gain-curves-pr-analysis-done-right.pdf

Improve this answer
$\endgroup$
3
  • $\begingroup$ I may have missed something, as I only skimmed the paper, but the precision-recall-gain doesn't seem to include true negatives in its calculations. Similar to a regular PR curve, won't this then be affected by the real-world (and therefore, in the context of the question, unknown) positive:negative data ratio? As the real-world data ratio is unknown, regular precision is (incorrectly) arbitrarily determined by the evaluation data ratio, which is why its not a good metric in this case. My initial reading of precision-gain suggests the same is true here. Please let me know if I'm mistaken. $\endgroup$
    – Shiania White
    Commented Nov 5, 2019 at 15:18
  • 1
    $\begingroup$ I think your mistaken, this isn't precision it is precision gain. The P/N ratio is used to standardise precision. The baseline is universal, the same as the ROC: people.cs.bris.ac.uk/~flach/PRGcurves//…. $\endgroup$
    – Christopher John
    Commented Nov 6, 2019 at 8:32
  • 1
    $\begingroup$ Thank you for the clarification. Once I have time, I'll investigate a bit deeper into the metric to see if it fits the other requirements from the question (intuitive-ness, not just resulting in a comparison of the number of 9s after the decimal point, etc). $\endgroup$
    – Shiania White
    Commented Nov 18, 2019 at 17:04
0
$\begingroup$

You may prefer phi coefficient, also known as MCC in machine learning community.

By the way, it may be worth saying that most meaningful metric is always the one that reflects the goal of your model, if you can specify it.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Could you explain why MCC might be more interpretable in this case than something like AUROC for this case of an unbalanced, but unknown data ratio? At first glance, I think this would trade one number very close to 1 for another number very close to 1. And though MCC makes for a good single value metric, it seems less interpretable than AUROC. That is, I have an understanding of what the difference between 0.99 and 0.999 AUROC intuitively means, but I have trouble concretely thinking about what 0.99 and 0.999 MCC means except that one is a little better. $\endgroup$
    – Shiania White
    Commented Nov 4, 2019 at 0:24
  • $\begingroup$ From your question there is no way to tell if phi will be that near to 1,one reason it is different is that it depends on contingency table, instead of predictions scores. Phi is a coefficient related to chi square, measuring how associated two variables are, given their marginal totals. Sure AUC has one more straightforward interpretation, at least. Anyway if your model is making that good predictions, every metric will give you extremes values. You can turn AUC in odds if you like it more that way. $\endgroup$
    – carlo
    Commented Nov 4, 2019 at 2:06
  • $\begingroup$ MCC is still an improper score that is dependent on the probability cut-off, 0.5 may not be correct. And also, I am not sure you can compare between different class distributions as it depends on the PPV and NPV. $\endgroup$
    – Christopher John
    Commented Nov 4, 2019 at 11:54
  • $\begingroup$ saying that a metric is improper because it depends on the cut-off, well that's improper. also precision and recall, that you suggested, depend on cut-off by the way. $\endgroup$
    – carlo
    Commented Nov 4, 2019 at 12:41
  • $\begingroup$ I didn't say it was improper because it is dependent on a cut-off actually. And PR gain curves like the ROC explore different thresholds of the system so are generally better. $\endgroup$
    – Christopher John
    Commented Nov 4, 2019 at 15:01
0
$\begingroup$

I would recommend proper scoring rules for the reasons I explain at Why is accuracy not the best measure for assessing classification models? These are admittedly not very intuitive. However, I have not yet come across a quality measure for classification that is both intuitive and not misleading, and I personally prefer a nonintuitive measure that does not steer me towards biased classifications, as do most of the published KPIs.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do you have any data to show proper scores like the Brier score are good for major class imbalance? I am interested in this. $\endgroup$
    – Christopher John
    Commented Nov 4, 2019 at 11:56
  • 1
    $\begingroup$ @ChristopherJohn: no, I don't have any data, sorry. You could always simulate to get a feeling. $\endgroup$
    – Stephan Kolassa
    Commented Nov 4, 2019 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.