1
$\begingroup$

Suppose you have a ML binary classifier that outputs the probability $p$ that a particular set of features belong to class $A$ (therefore the probability that the features are from class $B$ is $1-p$). Of course, those predictions don't take into account how well the classifier performed on validation sets, so I'm trying to figure out how to incorporate that information into the final result.

For example, I round the output probability up or down at 50% to make the classification. Suppose my classifier has a True Positive rate of 94% and a False Negative rate of 10% for class $A$ in a validation set. For a new, unseen feature set, it reports a probability of 98% that the features are from class $A$. That number should really go down, since there's a 6% chance it is a False Positive.

Strangely enough, I have yet to be able to find discussions online or in the literature about how to adjust the probability predictions to reflect validation findings. Or maybe I'm just looking in the wrong places...

$\endgroup$
5
  • 1
    $\begingroup$ Theres an inconsistency in your description. If your classifier reports a probability, it doen't inherently have a true positive rate or a true negative rate. So to make sense of the question, you'll have to explain that a bit more. When a classifier reports a probability for an event, as long as that classifier is well calibrated to the population, then that probability really is its best estimate of $P(y = 1 \mid X)$, there's no need to adjust it. $\endgroup$ Commented Jul 31, 2018 at 5:46
  • 1
    $\begingroup$ I round up or down at 50%. $\endgroup$
    – AstroBen
    Commented Jul 31, 2018 at 10:02
  • 1
    $\begingroup$ When you round up or down like that, you're being very destructive. You're replacing an estimate of a probability, which gives you lots of information about the classifiers degree of belief, and artificially forcing it to be completely confident. You shouldn't expect the outcome of such a destructive intervention to have much of anything to do with the quality of the original probabilistic estimates. I'll soapbox a bit: setting a classification threshold at 50%, and thinking the resulting rule tells you much of anything statistically is misguided.... $\endgroup$ Commented Jul 31, 2018 at 14:25
  • $\begingroup$ ...This is a fundamental misconception about classification. Models predict probabilities, and they come together with some measure of their uncertainty in the form of probabilities (as long as they are well calibrated). Hard classifiers do not, class assignment is a non-statistical issue. It should not be done lightly, and it should always be done in the context of the business or scientific problem you are trying to address. Every time someone naively thresholds a binary classifier at 50% they are making a serious error. $\endgroup$ Commented Jul 31, 2018 at 14:27
  • $\begingroup$ Matthew, I understand your points completely. What I had hoped was that by thresholding the probas on the validation results, I could establish a rough TN rate, and use that as a conservative prior to adjust output probabilities for new classifications--where I will report the proba, not a threaholded value. So I'm throwing away that validation info for a specific reason, not because I want to ignore it. Of course I'm happy to hear that this idea of mine is flawed. I've just found classifieds can report a 99% probability but they make sufficient errors that this isn't really believable. $\endgroup$
    – AstroBen
    Commented Jul 31, 2018 at 18:48

1 Answer 1

0
$\begingroup$

I found this blog post, which provides a convenient method and code for comparing the output prediction to empirical probability of a binary classifier, provided the validation set is sufficiently large.

http://jmetzen.github.io/2014-08-16/reliability-diagram.html

I fit a low-order polynomial to the result, which provides a reasonable conversion from classifier output to actual probability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.