We have sensitivity-specificity space (ROC curves) and precision-recall space (PR curves, $F_1$ score). What work has been done with PPV-NPV space?

Question

Receiver-operator characteristic (ROC) curves display the balance between sensitivity and specificity: how good you are at detecting category $1$ (sensitivity) while not falsely identifying category $0$ as category $1$ (specificity). This way, we can have some faith in the predicted $1$s really being $1$s.

Precision-recall (PR) curves arguably improve on this by giving a direct measure of the skepticism by using precision, which is a function of specificity (among other aspects of classifier performance). While recall/sensitivity measures the probability of detecting a $1$ given that an observation is a $1$, precision is the positive predictive value (PPV): the probability that your claimed $1$ really does belong to category $1$. Additionally, $F_1$ and more general $F_{\beta}$ scores consider both precision and recall.

Negative predictive value (NPV, not to be confused with net present value in finance) is to sensitivity/recall as positive predictive value is to specificity in that NPV gives the probability that your claimed $0$ really does belong to category $0$. Like PPV, NPV measures the forward flow of information by conditioning on the known prediction to estimate an unknown probability. It seems like comparing the PPV vs NPV would be valuable: how credible will our claims of category $1$ be for a given level of credibility of a claim of category $0$. This might not always be what we want to know, but it certainly could be.

What work is there that explicitly deals with PPV-NPV space, either a PPV-NPV curve or a classifier metric that combines PPV and NPV similar to how $F_{\beta}$ combines PPV/precision and recall/sensitivity? Sure, we can, along with the prevalence, transform a ROC curve into a PPV-NPV curve, but that does not mean that common ROC curves explicitly use the PPV and NPV.

Ideal would be a paper like Fawcett's An introduction to ROC analysis but for PPV-NPV curves.

REFERENCE

Fawcett, Tom. "An Introduction to ROC analysis." Pattern Recognition Letters 27.8 (2006): 861-874.

Answer 1

As you mentioned, some subsets of these metrics might be implicitly included in another chosen subset. Personally, I haven't come across much literature focused on the PPV-NPV space. I think it might be because it has some practical disadvantages.

One reason to prefer the ROC space (specificity $\operatorname{SPC}$, recall $\operatorname{REC}$) and the PR space (precision $\operatorname{PRE}$, recall $\operatorname{REC}$) over the PPV-NPV space is how they position a completely randomized algorithm and how they respond to prevalence (class imbalance). While extreme prevalences impact all prediction performance measures, in the ROC and PR spaces, at least one coordinate still places a random guess in the middle. This positioning accurately reflects that it is essentially a coin toss.

For example, consider a binary class with a true population prevalence $\eta \in (0,1)$ for the positive class and a classifier that randomly assigns the positive class with probability $p \in (0,1)$. Then: $$ \begin{aligned} \operatorname{REC} & = \frac{pP}{P} = p,\\ \operatorname{SPC} & = \frac{(1-p)N}{N} = 1-p,\\ \operatorname{PRE} & = \frac{pP}{pP+pN} = \frac{P}{P+N} =\eta,\\ \operatorname{PPV} & = \frac{\eta\operatorname{REC}}{\eta\operatorname{REC}+(1-\eta)(1-\operatorname{SPC})} = \frac{\eta p}{\eta p+(1-\eta)p} =\eta,\\ \operatorname{NPV} & = \frac{(1-\eta)\operatorname{SPC}}{(1-\eta)\operatorname{SPC}+\eta(1-\operatorname{REC})} = \frac{(1-\eta)(1-p)}{(1-\eta)(1-p)+\eta (1-p)} =1-\eta \end{aligned} $$

Now, let the classifier be a fair coin toss, so $p=0.50$, and consider a very low prevalence (high class imbalance), $\eta = 0.01$.

In the ROC space, we get $[\operatorname{SPC}=0.50; \operatorname{REC}=0.50]$, clearly indicating that the algorithm is just making random guesses. This location is an intrinsic evaluation of the algorithm, as it does not depend on the true prevalence. This is why, in the ROC space, the diagonal line represents the performance of a diagnostic test that is no better than chance.

In the PR space, we get $[\operatorname{PRE}=0.01; \operatorname{REC}=0.50]$. Here, one coordinate is independent of the prevalence and stays in the middle, partially indicating that the algorithm is making random guesses.

In the PPV-NPV space, we get $[\operatorname{PPV}=0.01; \operatorname{NPV}=0.99]$. Thus, a completely randomized algorithm merely reflects the prevalence, a measure that is not intrinsic to the algorithm but to the population distribution. Moreover, there is no fixed benchmark for "just chance" anymore.