6
$\begingroup$

What does predictive discrimination mean and how is it different from classification?

My question is prompted by Frank Harrell's comment:

Predictive discrimination is much more general a concept than classification.
...

This comment is at odds with my current thinking that predictive discrimination is simply an old-fashioned synonym for what is now called the supervised classification task. I would also assume that predictive discrimination is distinct from just discrimination, which could also mean clustering, depending on the context.

This term does not appear at all in books on machine learning that I usually use as a reference. Searching the web proved to be unproductive, as mostly sociopolitical applications of machine learning come up in the results.

$\endgroup$
4
  • 1
    $\begingroup$ +1. Let’s start simple: do you follow what Harrell means when he makes comments about how logistic regression is not a classifier despite many in machine learning considering it one? That is, do you understand what a logistic regression predicts? (It does not predict a category or a class, no matter what the predict method in sklearn gives.) $\endgroup$
    – Dave
    Commented Aug 22, 2022 at 3:30
  • $\begingroup$ @Dave Let's say, I distinguish between black-box classifiers (which return labels) and discriminant function-based models (which return a real number or several in the multi-class case), including those based on a probabilistic model, which return something like log-odds. Yet I tend to think of all of them as partitions of the data space into one or more regions corresponding to labels that would be assigned to inputs at the predict stage. $\endgroup$ Commented Aug 22, 2022 at 3:48
  • $\begingroup$ @Dave I couldn't find where exactly Frank Harrell says that logistic regression is not a classifier (or how it is not just a classifier). If missed something, please let me know. I can understand a derivation of logistic regression as a specialisation of the generalised linear model; logistic distribution feels a bit elusive to me, but also doable if needed. $\endgroup$ Commented Aug 22, 2022 at 4:08
  • 1
    $\begingroup$ I like two of his blog posts: (1) (2) $\endgroup$
    – Dave
    Commented Aug 22, 2022 at 4:18

2 Answers 2

2
$\begingroup$

Frank Harrell's comment [emphasis is mine]:

Predictive discrimination is the degree to which predictive signals can separate those with good outcomes from those with worse outcomes. The most popular measures of discrimination are R2 and c-index (concordance probability; equal to AUROC when Y is binary). Rank correlations between X and Y are measures of predictive discrimination. This is more general than classification as it takes into account tendencies/gray zones as in probability models. See https://fharrell.com/post/addvalue

His comparison of predictive discrimination to classification seems like apples and oranges to me, because the former is a statistic or performance measure, while the latter is a task or a problem setting. But this will do.

$\endgroup$
2
  • $\begingroup$ I’m surprised to see $R^2$ mentioned, since, at least in the binary setting (e.g., logistic regression), the usual $R^2$ is a function of the Brier score that also considers calibration. Perhaps Harrell would calculate $R^2$ as $\left(\operatorname{corr}\left(y,\hat y\right)\right)^2$, which can be equal to $1-\frac{SSRes}{SSTotal}$ in some situations (such as in-sample OLS) but not necessarily in others. (The $SSRes$ is related to the Brier score when the model outputs are probabilities (e.g., logistic regression).) $\endgroup$
    – Dave
    Commented Jan 28, 2023 at 20:43
  • $\begingroup$ @Dave Frank explains how he calculates $R^2$ in the linked post under the section "Key Measures". I think he uses Nagelkerke's $R^2$ frequently. $\endgroup$ Commented Jan 28, 2023 at 21:23
2
$\begingroup$

Predictive discrimination is the ability of a model to produce (distributions of) predicted values that are separated when the observed values are distinct, and to have the correct order, and I think this is totally consistent with the quote in the answer by paperskilltrees. This is not the same as making quality predictions. A model can have good, even perfect, predictive discrimination while having terrible performance, as I will demonstrate below.

set.seed(2023)
N <- 1000
y_true <- runif(N, 0, 1)
y_pred <- 10 + y_true
plot(y_true, y_pred)

enter image description here

The predicted values are perfectly correlated with the true values, yet the predictions are terrible.

Similarly, a model that predicts probabilities (a "classifier" in many circles) can produce probabilities that are well-separated between the classes yet are not reflective of the true probability of an event occurring. This would be reflected in the ROCAUC being high (or the correlation between the true and predicted values) yet the performance according to something like Brier score or log loss being poor.

library(rms)
set.seed(2023)
N <- 1000
p_true <- rbeta(N, 1/4, 1/4)
y_true <- rbinom(N, 1, p_true)
y_pred <- ecdf(p_true)(p_true)
rms::val.prob(y_pred, y_true)

plot of actual probability versus predicted probability

The measures of predictive discrimination, AUC and squared correlation, are quite high (especially AUC), but the calibration is terrible, as the curved plot that deviates from the "ideal" shows.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.