2
$\begingroup$

How can I optimize training for a neural network so that I can get a low false positive ratio? I'd like to have a binary classifier that recognizes some positive examples very reliably, that is with very low false positive ratio, but don't care much about the false negative ratio (even 99.9% false negatives would be OK.) Is there a special loss function that is appropriate for this scenario? Special neural network structures / training regimes? Are there some keywords I could search for, to find more information what's appropriate for such a scenario?

Update: With this question I'm wondering whether mostly dropping the requirement of a reasonable false negative ratio leads to new avenues that wouldn't make sense otherwise. For instance, if there was an algorithm that would find tiny islands of easily positively classified examples in a sea of not so easily classified examples, that'd perfectly fit the question.

And please note that I'm talking about a low false positive ratio, not about few false positives, which excludes the degenerate "classify everything as negative" classifier from consideration.

Intuition: To give you a reason for my hunch that there is something special about that scenario: the usual simple neural network trained as a classifier would more or less be trained for a good classification performance over all examples. But with a radial basis function network you could in principle pick out a few interesting areas where most examples are positive, discarding all the rest, though I wouldn't know how to train it like that. So there could be special network structures, special ways to train, special loss functions and whatnot that are suited to this kind of problem.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Why not just predict that all cases are a non-event? With such a prediction, your false positive is always 0 (because you never predict a positive class, hence you can never make a false positive) and your false negative is very close to 99%. $\endgroup$
    – Demetri Pananos
    Commented Sep 4, 2022 at 15:53
  • $\begingroup$ @DemetriPananos I said that the false negative ratio is almost irrelevant, not completely irrelevant. :-) I do want some positive classifications, but it's OK if they are relatively few where the classification is pretty sure. Also, your suggestion would get 0/0 as a false positive ratio, which is not low but just undefined, so that doesn't help. $\endgroup$
    – Dr. Hans-Peter Störr
    Commented Sep 5, 2022 at 20:54
  • 1
    $\begingroup$ Well, the count of false positives exists -- its 0 -- the problem is just that you've made no positive predictions, which is fine to me. But this is besides the point. Your needs are a bit vague ("not completely irrelevant" is understandable, but not something we can work with). Stephan's answer is perhaps the best way forward, but you're still going to need to associate costs to a false positive and false negative. From there, the optimal cutoffs for classification can be computed. $\endgroup$
    – Demetri Pananos
    Commented Sep 5, 2022 at 21:47
  • $\begingroup$ @DemetriPananos Still, I deliberately asked for a low false positive ratio, not for few false positives, which excludes the degenerate "everything is false" classifier. I updated the question - hopefully the intention is clearer now; I want to keep that point vague to keep it general. I guess in most applications you'd want to have a reasonably low false negative ratio - if the classifier had 99.9% there, it'd immediately be thrown away. But I'd go into that corner. $\endgroup$
    – Dr. Hans-Peter Störr
    Commented Sep 6, 2022 at 20:02

1 Answer 1

Reset to default
4
$\begingroup$

Train a probabilistic classifier with a proper scoring rule as a loss. If you then want a hard 0-1 classification (which I would argue is not part of the statistical modeling step, but of the subsequent decision step), use a threshold that is high enough for your purposes. Yes, that may result in a threshold that is so high that everything is classified as non-target - which will give you a zero FPR. If this does not address your concern, I would recommend thinking about whether your hard 0-1 classification is really what you want to do.

More information here. For more information about the problems with accuracy, FPR, FNR see Why is accuracy not the best measure for assessing classification models?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for your good hints and blog entries! Interestingly, I had tried something like that. But my problem always was that when the treshold is so high that only relatively few examples of the test set match, almost always the FPR actually went down the higher the treshold gets, instead of going further up. Possibly because there were so few training examples like that, so that the network is over-fitting. So I was wondering if there is something specially geared to that kind of problem. (By the way: if everything is classified as non-target, the FPR is 0/0 = undefined, not zero. :-) $\endgroup$
    – Dr. Hans-Peter Störr
    Commented Sep 6, 2022 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.