3
$\begingroup$

Context: I am working on a predictive model. Let's call it $f$. The outcome that $f$ is trying to predict is binary. It makes predictions as probabilities, i.e. for a given input $x$, $f(x) \in (0,1)$. Its performance seems to be so-so: The AUC is 0.67 and the average precision is 0.73 (compared to 58% positive cases in the test set).

I was pondering the merits of the model when I came up with this approach at assessing the model's performance that (I think/hope) takes noise and inherent uncertainty in the data into account:

grouped predicted probability against proportion of positive cases

I'll explain this table by explaining what the row with index (0.2, 0.3] says: Of the model's predictions on the test set, 30 predictions were between 0.2 and 0.3. Of these 30 points, 5 were positive (outcome = 1). The proportion 0.1667 is 5 divided by 30.

The idea behind this approach is frequentist in nature: If the model makes predictions of 0.25 for a number of points, then 25% of these points should have a positive outcome. Likewise for other probabilities. In this particular case the proportion does indeed lie in the given range for most of the rows. So I guess the model is actually okay?

I have tried to find this approach somewhere in the literature, but it's hard to know which keywords to use (see question 2 below).

My question is:

  1. Is this a meaningful approach?

If it is a meaningful approach, then two more questions:

  1. Does it have a name?
  2. Is it really just the ROC curve, the precision–recall curve or even the log loss in (discrete) disguise?
$\endgroup$

1 Answer 1

5
$\begingroup$

If you plot the binned proportions of positive cases against the bins, you get a reliability diagram. We have a tag which is not really helpful in this case, and a number of relevant threads. Dimitriadis et al. (2021) is a recent treatment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.