2
$\begingroup$

I'm looking for a metric that behaves like this

enter image description here

Here, N can be interpreted as the total number of positive predictions. (Alternatively, you can interpret N as the total number of true positives. Either should be fine.)

I want the score to fall off a cliff if the false discovery rate goes above 5%. Otherwise, the metric should strive for the highest number of positive classifications.

Does such a classification metric exist?

Motivational Example

Suppose I'm building a computer vision model to identify shoplifters. I want the ability to promise that the model will flag as many shoplifters as possible without exceeding a 5% false discovery rate.

$\endgroup$
8
  • $\begingroup$ Your chart seems to suggest that identifying $50$ false positives and $0$ true positives (assuming there are over $1000$ non-shoplifters) would be better than $0$ false positives and $10$ true positives $\endgroup$
    – Henry
    Commented Nov 22, 2022 at 20:06
  • $\begingroup$ Oh shoot. I meant to say "False Discovery Rate" not "False Positive Rate". Fixing this now.. sorry! $\endgroup$
    – Ben
    Commented Nov 22, 2022 at 20:13
  • $\begingroup$ Why not use $N \cdot \textbf{1}_{FPR<0.05}$? $\endgroup$ Commented Nov 28, 2022 at 19:47
  • $\begingroup$ @SextusEmpiricus I don't want something quite so rigid. If my model ends up with a FDR of 6%, it's not a big deal. I just don't want it to end up with a FDR of 50% simply to catch more true positive instances. $\endgroup$
    – Ben
    Commented Nov 28, 2022 at 23:05
  • $\begingroup$ There are all sorts of ways to have a cost function $f(FPR,TPR)$ peak around 5%. How do you want it exactly if the indicator function is not right? Maybe it could be better if you tell more about the way that the true positive rate should play a role and and what sort of comparisons you make that require some fancy function (why not just select the model that is best at FPR 0.05?). How exactly is the cost in practice when you get over 0.05? What is the ratio of positive and negative cases? How should the model scale for different of such ratio's. How do you wheigh the cost of a false negative? $\endgroup$ Commented Nov 29, 2022 at 7:32

1 Answer 1

1
+100
$\begingroup$

You could consider incorporating the classification threshold in the optimization of TPR (along with the usual parameter estimates) and applying a constraint that FPR be less than $5\%$. This would be something like a Karush-Kuhn-Tucker constrained optimization (maximize TPR, constrained to FPR no greater than a particular rate like $5\%$), though the function being optimized will have jumps rather than being smooth. This gets rid of the richness of probabilistic predictions but might be your best bet if you have a hard cutoff of $5\%$ FPR. Then your metric would be TPR, given FPR no greater than the maximum allowed. (If you allow for parameter tuning at the same time as threshold tuning, my suspicion is that you can hit exactly $5\%$ FPR, at least in theory.)

This is evocative of hypothesis test theory where we consider the power (similar to TPR) of a test at a given tolerance for type I errors (false positives).

ORIGINAL

Your comments indicate that $5\%$ FDR is not such a hard requirement. Consequently, it is not quite correct to say that you want to optimize TPR without exceeding $5\%$ FDR.

It sounds like you want accurate predictions of positive/negative probability, which is exactly what the standard Crossentropy loss measures: lower loss means better predictions of the positive/negative probability. Crossentropy loss is one example of a so-called “strictly proper scoring rule”.

Once you have accurate probabilities, you can fiddle with thresholds to get your desired tradeoff of FPR and TPR, which might turn out to mean that you tolerate $6\%$ or $8\%$ FPR because of how much of an improvement in TPR you get by accepting a slightly higher FPR. Likewise, you might find yourself only accepting $3\%$ FPR due to how little improvement you get in TPR upon tolerating slightly higher FPR; you might have to go up to $20\%$ FPR to get much improvement in TPR.

Note that there can be multiple actions despite there being only two observed categories. That is one place where predicting probabilities rather than discrete classes gives a lot of power. It’s good to be able to say, “I don’t know, and making either kind of wrong classification is pretty costly, so I’d like to gather more information.” We prefer when models can be confident in their predictions, sure, but that isn’t always realistic.

EDIT

Models tend to output values in a continuum, frequently bounded between $0$ and $1$ and interpreted as probabilities. A common, maybe even ubiquitous, software default is to use $0.5$ as a threshold for binning the continuous values into discrete categories. For a software default, if there must be one, I suppose it makes sense to classify the observation according to the class to which it most probably belongs. However, there is a cost associated with wrong decisions, and you might be willing to sacrifice classifications of a not-so-important category in order to catch the category of primary interest. As an example from medicine, it is inconvenient to have to go into a Covid quarantine, but society seemed to deem it reasonable to put non-Covid cases in quarantine to assure that very few true Covid cases got missed and wandered about the world to infect others.

(You are free to disagree with how Covid quarantines and lockdowns occurred, but many societies did take this stance. That just means you would prefer a different cutoff than society seemed to adopt. Likewise, if you think society didn’t do enough and should have been in a near-$100\%$ lockdown, it just means you would have preferred a different threshold and much lower evidence of Covid in order to warrant isolation.)

Thus, we might want to change the threshold away from $0.5$ in order to suit our values about the decisions. Maybe, instead of requiring high evidence of a category being likely, we want to flag as soon as the probability is above $0.1$. Then $0.1$ would be our threshold instead of $0.5$, and we would get the TPR and FPR rates that go along with such a threshold.

By fiddling with the threshold, you can tune your FPR and TPR.

Let’s look at a simulation where I do some tuning of the threshold.

library(MLmetrics)
set.seed(2022)
N <- 100
p <- sort(runif(N, 0, 1))
y_true <- rbinom(N, 1, p)
p_pred <- sort(rbeta(N, 1/3, 1/3))

# Let's use the default threshold of 0.5
y_pred <- round(p_pred)
tpr <- MLmetrics::Sensitivity(y_true, y_pred)
fpr <- 1- MLmetrics::Specificity(y_true, y_pred)
print(paste(
  "Threshold: ", 0.5
  "TPR:", tpr, 
  "; FPR,", fpr
))

# Let's use a threshold of 0.1 = 0.5 - 0.4
# There's a better way of doing this than subtracting from the 
# predicted probabilities, but I can't remember the syntax
#
y_pred <- round(p_pred - 0.4)
tpr <- MLmetrics::Sensitivity(y_true, y_pred)
fpr <- 1- MLmetrics::Specificity(y_true, y_pred)
print(paste(
  "Threshold: ", 0.1,
  "TPR:", tpr, 
  "; FPR,", fpr
))

# Let's use a threshold of 0.9 = 0.5 + 0.4
#
y_pred <- round(p_pred + 0.4)
tpr <- MLmetrics::Sensitivity(y_true, y_pred)
fpr <- 1- MLmetrics::Specificity(y_true, y_pred)
print(paste(
  "Threshold: ", 0.9,
  "TPR:", tpr, 
  "; FPR,", fpr
))

# Let's use a threshold of 0.99 = 0.5 + 0.49
#
y_pred <- round(p_pred + 0.49)
tpr <- MLmetrics::Sensitivity(y_true, y_pred)
fpr <- 1- MLmetrics::Specificity(y_true, y_pred)
print(paste(
  "Threshold: ", 0.99,
  "TPR:", tpr, 
  "; FPR,", fpr
))

# Let's use a threshold of 0.95 = 0.5 + 0.45
#
y_pred <- round(p_pred + 0.45)
tpr <- MLmetrics::Sensitivity(y_true, y_pred)
fpr <- 1- MLmetrics::Specificity(y_true, y_pred)
print(paste(
  "Threshold: ", 0.95,
  "TPR:", tpr, 
  "; FPR,", fpr
))

I get that your ideal threshold is between $0.95$ and $0.99$. You can fiddle with this to your heart's content to get the threshold that works best for you, noting the way the threshold impacts both FPR and TPR. If, in your analysis of this, you find yourself getting a huge increase in TPR by accepting an FPR of $0.07$ instead of $0.05$, it is your perogative to accept that.

This is related to, but not quite the same as, looking at ROC curves. Key differences are:

  1. ROC curves have nothing to do with calibration, and you can divide all of your probabilities by two and wind up with the same ROC curve.

  2. ROC curves can involve TPR and FPR values that you would never accept.

Frank Harrell has written negatively of ROC curves. He has this amusing tweet and multiple mentions of his dislike of ROC curves in his blog and posts on Cross Validated. Among his complaints are that TPR and FPR are not aligned with forward-looking decision-making, and you might consider looking and forward-looking metrics like positive predictive value and negative predictive value. However, if you are determined to work with TPR and FPR, then that’s what you have to work with.

If you do choose to go with positive and negative predictive values, you also can tune the threshold as I have done in order to get your desired tradeoff. After all, you have to create the categories out of the probability outputs somehow, and there is no reason why the software default must be the appropriate threshold for your task.

Note that tuning the threshold and forcing every observation to be in a discrete category gets rid of your ability to have a grey zone of “I don’t know, collect more data,” which might be the best decision in such a situation.

$\endgroup$
7
  • $\begingroup$ Thanks. Can you elaborate on "Once you have accurate probabilities, you can fiddle with thresholds to get your desired tradeoff of FPR and TPR"? Keep in mind that I am still striving for 5% FDR. What thresholds can I tune to achieve this and how? Perhaps a concrete example would help. $\endgroup$
    – Ben
    Commented Dec 5, 2022 at 17:01
  • $\begingroup$ @Ben, is this some sort of a homework question? You are not asking this for a concrete example that you have yourself? $\endgroup$ Commented Dec 5, 2022 at 17:57
  • $\begingroup$ @Ben Hopefully my edit clarifies. $\endgroup$
    – Dave
    Commented Dec 5, 2022 at 18:06
  • $\begingroup$ @SextusEmpiricus not a homework question. This is for my work, but I can't share the exact data I'm working with. $\endgroup$
    – Ben
    Commented Dec 5, 2022 at 19:08
  • $\begingroup$ @Dave Thanks for writing up this example. I very familiar with the concepts you mention in your post, but it looks like you have not actually proposed an evaluation metric - merely a system for picking a classification threshold. (The best metric proposed thus far is $𝑁⋅1_{𝐹𝑃𝑅<0.05}$ by @SextusEmpiricus) $\endgroup$
    – Ben
    Commented Dec 5, 2022 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.