We have sensitivity-specificity space (ROC curves) and precision-recall space (PR curves, $F_1$ score). What work has been done with PPV-NPV space?

Receiver-operator characteristic (ROC) curves display the balance between sensitivity and specificity: how good you are at detecting category $$1$$ (sensitivity) while not falsely identifying category $$0$$ as category $$1$$ (specificity). This way, we can have some faith in the predicted $$1$$s really being $$1$$s.

Precision-recall (PR) curves arguably improve on this by giving a direct measure of the skepticism by using precision, which is a function of specificity (among other aspects of classifier performance). While recall/sensitivity measures the probability of detecting a $$1$$ given that an observation is a $$1$$, precision is the positive predictive value (PPV): the probability that your claimed $$1$$ really does belong to category $$1$$. Additionally, $$F_1$$ and more general $$F_{\beta}$$ scores consider both precision and recall.

Negative predictive value (NPV, not to be confused with net present value in finance) is to sensitivity/recall as positive predictive value is to specificity in that NPV gives the probability that your claimed $$0$$ really does belong to category $$0$$. Like PPV, NPV measures the forward flow of information by conditioning on the known prediction to estimate an unknown probability. It seems like comparing the PPV vs NPV would be valuable: how credible will our claims of category $$1$$ be for a given level of credibility of a claim of category $$0$$. This might not always be what we want to know, but it certainly could be.

What work is there that explicitly deals with PPV-NPV space, either a PPV-NPV curve or a classifier metric that combines PPV and NPV similar to how $$F_{\beta}$$ combines PPV/precision and recall/sensitivity? Sure, we can, along with the prevalence, transform a ROC curve into a PPV-NPV curve, but that does not mean that common ROC curves explicitly use the PPV and NPV.

Ideal would be a paper like Fawcett's An introduction to ROC analysis but for PPV-NPV curves.

REFERENCE

Fawcett, Tom. "An Introduction to ROC analysis." Pattern Recognition Letters 27.8 (2006): 861-874.

As you mentioned, some subsets of these metrics might be implicitly included in another chosen subset. Personally, I haven't come across much literature focused on the PPV-NPV space. I think it might be because it has some practical disadvantages.

One reason to prefer the ROC space (specificity $$\operatorname{SPC}$$, recall $$\operatorname{REC}$$) and the PR space (precision $$\operatorname{PRE}$$, recall $$\operatorname{REC}$$) over the PPV-NPV space is how they position a completely randomized algorithm and how they respond to prevalence (class imbalance). While extreme prevalences impact all prediction performance measures, in the ROC and PR spaces, at least one coordinate still places a random guess in the middle. This positioning accurately reflects that it is essentially a coin toss.

For example, consider a binary class with a true population prevalence $$\eta \in (0,1)$$ for the positive class and a classifier that randomly assigns the positive class with probability $$p \in (0,1)$$. Then: \begin{aligned} \operatorname{REC} & = \frac{pP}{P} = p,\\ \operatorname{SPC} & = \frac{(1-p)N}{N} = 1-p,\\ \operatorname{PRE} & = \frac{pP}{pP+pN} = \frac{P}{P+N} =\eta,\\ \operatorname{PPV} & = \frac{\eta\operatorname{REC}}{\eta\operatorname{REC}+(1-\eta)(1-\operatorname{SPC})} = \frac{\eta p}{\eta p+(1-\eta)p} =\eta,\\ \operatorname{NPV} & = \frac{(1-\eta)\operatorname{SPC}}{(1-\eta)\operatorname{SPC}+\eta(1-\operatorname{REC})} = \frac{(1-\eta)(1-p)}{(1-\eta)(1-p)+\eta (1-p)} =1-\eta \end{aligned}

Now, let the classifier be a fair coin toss, so $$p=0.50$$, and consider a very low prevalence (high class imbalance), $$\eta = 0.01$$.

In the ROC space, we get $$[\operatorname{SPC}=0.50; \operatorname{REC}=0.50]$$, clearly indicating that the algorithm is just making random guesses. This location is an intrinsic evaluation of the algorithm, as it does not depend on the true prevalence. This is why, in the ROC space, the diagonal line represents the performance of a diagnostic test that is no better than chance.

In the PR space, we get $$[\operatorname{PRE}=0.01; \operatorname{REC}=0.50]$$. Here, one coordinate is independent of the prevalence and stays in the middle, partially indicating that the algorithm is making random guesses.

In the PPV-NPV space, we get $$[\operatorname{PPV}=0.01; \operatorname{NPV}=0.99]$$. Thus, a completely randomized algorithm merely reflects the prevalence, a measure that is not intrinsic to the algorithm but to the population distribution. Moreover, there is no fixed benchmark for "just chance" anymore.

• I think there is a benchmark for "just chance" in that PPV is the prevalence and NPV is one minus the prevalence. // PR space even reflects the data instead of just the model quality. In fact, so do log loss and Brier score (inherently easier to get a lower score when the "just chance" predictions are usually close). I'm not sure I buy that as an argument against looking at PPV-NPV space.
– Dave
Commented Jun 5 at 14:01
• Any small subset of these measures (in low-dimensional spaces) would be incomplete in terms of capturing all properties and interpretations, though some might be sufficient for a specific application. For me, a benchmark for "just chance" should be fixed to facilitate easier comparability. In the PPV-NPV space, we do not get that fixed benchmark, as it merely reflects the prevalence, which is not observed and must be estimated from finite samples. What happens with the "just chance" benchmark under distribution shift, for instance? Commented Jun 5 at 14:12
• The "just chance" benchmark must be estimated for PRAUC, log loss, Brier score, classification accuracy, square loss, absolute loss, pinball loss, and many other common measures of performance. The only "just chance" benchmark I know that has a constant value is the ROCAUC. The issue of distribution shift is an interesting matter, but that does not seem like a legitimate criticism of PPV-NPV space when it applies all over the place.
– Dave
Commented Jun 5 at 14:24