Binary metric for an unknown, but highly imbalanced, data ratio? I'm looking for a good metric to compare binary classification methods for a task where


*

*The data is highly imbalanced.

*The approximate data imbalance is unknown.


There are certainly more than 100 negative examples for every positive one. However, how much more is unknown. It may be 1:1000 or 1:100000 or more.
In this situation, precision doesn't seem to make sense to use as a metric, because we don't know what the real imbalance is and precision will change depending on that ratio. ROC values (true positive rate and false positive rate) have a real meaning regardless of the ratio. However, the AUROC is very close to 1 and the ROC curve approaches looking like a 90 degree corner. Comparing an AUROC of 0.999 and 0.9999 isn't very intuitive.
Is there a metric for such a situation that allows for an intuitive comparison between different models?
 A: Yes, there is the precision-recall gain curve which is for severely imbalanced data where the class distribution can vary between tasks and you want to compare performance between them. It is straightforward to implement and published in NIPS. It works by standardising precision to the baseline chance expectation. This is for where you care about the positive class, but not negative.
https://papers.nips.cc/paper/5867-precision-recall-gain-curves-pr-analysis-done-right.pdf
A: You may prefer phi coefficient, also known as MCC in machine learning community.
By the way, it may be worth saying that most meaningful metric is always the one that reflects the goal of your model, if you can specify it.
A: I would recommend proper scoring rules for the reasons I explain at Why is accuracy not the best measure for assessing classification models? These are admittedly not very intuitive. However, I have not yet come across a quality measure for classification that is both intuitive and not misleading, and I personally prefer a nonintuitive measure that does not steer me towards biased classifications, as do most of the published KPIs.
