3
$\begingroup$

If we have a true classifier, it can make sense to calculate measures of performance like accuracy, precision (positive predictive value), and recall (sensitivity). Each of these has something to do with the classifier output being exactly correct or totally incorrect, and we would hope for a classifier to be exactly correct in most instances.

In regressions, we don't expect our predictions to be spot on, and we typically measure model performance using measures like mean squared error or mean absolute error that give penalties for any deviations from the observed value but decrease the penalty as the (slightly incorrect) predictions get closer to the observations.

It might seem like regression metrics and classification metrics are fundamentally different, as the latter expect exactly correct predictions and the former can report solid performance despite every prediction being at least a little bit incorrect. However, consider the following.

I have a predictor of financial returns, and it makes a prediction of a large decrease in value. If my predictions are just "big loss" vs "business as usual", then I can calculate the positive predictive value of a "big loss" prediction: the probability that there will be a big loss, given that such a prediction has been made. If the positive predictive value is low, then, despite the prediction of a big loss, a big loss is not likely. It makes sense to me that this can be extended to a continuum: given a predicted loss of $90\%$ (I'd say that counts as a big loss), what is the probability of a loss of at least $90\%?$ I think this can be calculated by taking the number of times a loss of at least $90\%$ was predicted and looking at how many times the loss was at least as great as predicted (though this seems like a worse idea after typing it than it did when I first thought of it). However, what about if the loss is $89\%?$ Such a situation is completely ignored by that method, yet I would be alarmed by such a loss and would not consider my predictor to have made such a mistake if it predicts a loss of $90\%$ and then a loss of $89\%$ occurred.

So is there a way to extend notions like positive predictive value to predictions made on a continuum?

In math, positive predictive value is:

$$ \text{PPV} = P( Y = 1\mid \hat Y = 1) $$

Perhaps I mean to ask about calculating or estimating the density $f_{Y\vert \hat Y = 90\%}$.

(This comes from a discussion in the comments to this CV question where a member wants a model to make more extreme predictions, even if that means most of them will occur when more modest outcomes are observed, and I retort that then the model's extreme predictions would not be credible, much as a classifier of an unusual event would not be credible (low positive predictive value) if it typically made such predictions yet those predictions turned out to be false. In both cases, the predictor would often "cry wolf". In the case discussed at the link, that density I suspect I want to estimate would be something like $f_{Y\vert \hat Y = \text{Extreme}}$, where most of the density is nowhere near that "extreme" $\hat Y$.)

$\endgroup$

1 Answer 1

2
$\begingroup$

YES

In classification, precision is the probability of an event being positive conditioned on the event being predicted to be positive. This is just a Bernoulli distribution with a certain success probability, defined by the precision. In other words, the distribution of the outcome $Y$, conditioned on the prediction being positive, is $\text{Bernoulli}(\text{precision})$. Therefore, the interest in precision is the distribution of the outcome, conditioned on the prediction being a certain value.

Thus, a precision analogue in regression would be the distribution of the outcome, conditioned on the prediction being a certain value: $F_{Y\vert \hat Y = \hat y}\left(y\right)$, perhaps better written as $F\left(y; Y\vert Y = \hat y\right)$ because of the slightly complicated random variable subscript.

$F_{Y\vert \hat Y = \hat y}\left(y\right)$ is exactly the distribution of the outcome $Y$ conditioned on the prediction $\hat Y$ equaling a value $\hat y$. When we are in a classification setting where we want to know the distribution of the binary $Y$ when $\hat Y$ is the positive prediction $1$, then $F_{Y\vert \hat Y = \hat y}\left(y\right) = F_{Y\vert \hat Y = 1}\left(y\right)$ is the CDF of a Bernoulli variable with precision as the probability parameter.

Therefore, the generalization of precision is $F_{Y\vert \hat Y = \hat y}\left(y\right)$. This is the distribution of values of $Y$ when the prediction equals a particular quantity, $\hat y$.

This rationale can be applied to sensitivity, too. If sensitivity has to do with the distribution of predictions conditioned on the truth being a particular value, the general expression of this is $F_{\hat Y\vert Y = y}$, where $Y = 1$ gives the desired distribution for sensitivity.

Since there is no notion of "positive" or "negative" category in a general regression problem, negative predictive value and specificity also generalize to $F_{Y\vert \hat Y = \hat y}\left(y\right)$ and $F_{\hat Y\vert Y = y}\left(y\right)$, respectively. The former is the distribution of true values, conditioned on a prediction (what negative and positive predictive values concern), while the latter is the distribution of predicted values, conditioned on a true value (what specificity and sensitivity concern).

(I suppose the $F$ could be lowercase if we are willing to take the density $f$ to be a Radon-Nikodym derivative that allows for both continuous and categorical distributions (among others), but I like the idea of being able to generalize through the CDF without having to appeal to measure theory.)

$\endgroup$
12
  • $\begingroup$ I'm still at a loss for how to calculate or estimate these distributions, however, so I am open to answers from others! $\endgroup$
    – Dave
    Dec 21, 2023 at 21:02
  • $\begingroup$ Starting with classification as a goal leads to inefficiency and confusion. It leads analyzes to use accuracy measures that are easily gamed and lead to selection of the wrong model. Details are here. Cast the problem as a risk estimation problem and not only will better decisions result, but many of the confusing problems just vanish. $\endgroup$ Dec 22, 2023 at 9:31
  • $\begingroup$ @FrankHarrell So then what’s the distribution of observed values when my predicted risk is an event probability of $0.2?$ $\endgroup$
    – Dave
    Dec 22, 2023 at 13:25
  • $\begingroup$ Start with the Brier score which doesn’t need predicted values to be coarsened into positives and negatives. $\endgroup$ Dec 22, 2023 at 15:38
  • $\begingroup$ @FrankHarrell But what about when the prediction is $0.2?$ There is no sense of a "positive" or "negative" prediction in such a situation. $\endgroup$
    – Dave
    Dec 22, 2023 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.