0
$\begingroup$

I have a binary classification model whose ROC curve looks like the one below. The black point is the optimal probability threshold to use by calculating the geometric mean. However, that's a pretty high false-positive rate and something around the red point would be more optimal for my use case. I was wondering if there are any ways to choose the optimal threshold with the same technique of geometric mean but assigning weights to your true and false positive rates depending on its importance. e.g. I give 0.8 importance to false positives and 0.2 to true positives meaning that I penalize having false positives.

Roc cruve

$\endgroup$
1
$\begingroup$

You need to define a cost function

Perkins & Schisterman (2006) propose a pretty straigtforward approach using the relative cost of misclassifications that I would recommend as a starting point. They start from the Youden Index $J$

$$J=max\{q(c)+p(c)−1\}$$

and update it with the relative cost $r$

$$J=max\{q(c)+r×p(c)−1\}$$

with $r = (1 − π)/aπ$ where $π$ is the prevalence and $a$ is the relative cost of a false-negative classification as compared with a false-positive classification

$$a = \frac{\text{Cost of a false negative}}{\text{Cost of a false positive}}$$

There are several other methods, and López-Ratón et al (2014) provide a pretty good review.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ What is the $\alpha$ relative cost exactly?. I went through the paper but it seems there is no specific explanation for this. For instance, I have 21 positive cases and 4046 negative ones. Prevalence is going to be 21/4046 = 0.005. I changed $p(c)$ to $1 - fpr$ where fpr is false positive rate. I want false positives to have a higher importance so I tried with $\alpha$ values 0.7 and 3. I get a threshold greater than 1. $\endgroup$ – Brandon Apr 6 at 13:42
  • $\begingroup$ @Brandon I clarified my answer to spell out the relative cost. $\endgroup$ – Calimo Apr 8 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.