11
$\begingroup$

I'm working on a task that even a 0.00001 fp rate is not acceptable, because detecting something as a positive when its not will have very bad consequences in this task, so it needs to be exactly 0 in my dataset when i use k fold, so 0 for each fold. basically my model should at least learn all the negative samples in my own dataset very well and never classify them as a positive by mistake.

but what is the best way of doing this?

so far two things came to my mind but please let me know if there is a better method :

  1. Giving positive samples a very large weight during training

  2. Data augmentation of positive samples, so making the positive dataset 100 time bigger or something

to sum up the question :

You are giving a binary classification task with enough balanced data, and are asked to train a deep neural model with 0 false positive rate on the given dataset, how will you do it? (input dim is around 1k-3k)

Improve this question
$\endgroup$
5
  • 11
    $\begingroup$ The only way to guarantee 0 FPR is to not declare anything as a positive. Otherwise, you'll have to tolerate some amount of FPR. $\endgroup$
    – Sycorax ♦
    Oct 16, 2020 at 5:05
  • $\begingroup$ @Sycorax i am already getting 0 FPR by giving high weight to positives, but that causes accuracy to go down 20% - 25%. the dataset is around 10 million so its not because of lack of data either. I'm just looking for best method to do this so the accuracy drops as little as possible. $\endgroup$
    – OneAndOnly
    Oct 16, 2020 at 5:31
  • 3
    $\begingroup$ In practice, one has no assurance of zero false positives if any true positives are identified. I would suggest balancing the costs for false positives and false negatives, to avoid this and ask for a zero means you do not care at all that false negatives occur, which from the context in which you frame the question is not true, you obviously do care about false negatives. For example, if 100 false negatives costs as much as one false positive, I would set the rates accordingly; not at zero, but at 1/100. $\endgroup$
    – Carl
    Oct 16, 2020 at 6:10
  • $\begingroup$ @Carl yes i know that in real world implementation there will be false positives no matter how hard i try and there is no assurance for that, but i need to bring FPR very close to 0 in real world and exactly 0 in my own dataset ( i am using k fold and dont want to have any false positive in any fold ). so you are suggesting that the best way to do this is giving high weights to positive samples correct? $\endgroup$
    – OneAndOnly
    Oct 16, 2020 at 7:59
  • $\begingroup$ The best way to do things is with realistic estimates, not absolutist ones. Asking for zero FPR has an answer of 100% false negatives for a population (where a population is defined as an infinite number of realizations). Typically, a more cost balanced approach will yield the minimum cost of errors for the same reason that the maximum product of a constant sum occurs when the two numbers being summed are equal. $\endgroup$
    – Carl
    Oct 16, 2020 at 22:52

2 Answers 2

Reset to default
14
$\begingroup$

Use probabilistic classifications instead of hard 0-1 classifications. That is, predict the probability for an instance to be positive. Use proper scoring rules to assess these predicted probabilities.

Then consider whether you can make decisions based on these probabilities. You may or may not want to use a single threshold to map your probabilities to hard classes. Instead, you may even want to use multiple thresholds for multiple different actions. The mapping between probabilities and decisions should be based on explicit assumptions about the costs of wrong (and correct) decisions. More here.

In a nutshell: decouple the modeling/predictive part from the decision.

Do not use accuracy as a KPI at all. It is misleading, and especially (but not only) so for unbalanced data. The exact same problems as for accuracy apply equally to FPR.

Similarly, do not overweight one class. This is analogous to oversampling, which is commonly used to "address" class imbalance - but unbalanced data are not a problem (as long as you don't use misleading KPIs like accuracy or FPR), and oversampling or weighting will not solve a non-problem.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ If possible interest to readers: stats.stackexchange.com/questions/464636/…. The guy who answered might have a familiar-looking name. $\endgroup$
    – Dave
    Oct 16, 2020 at 11:55
  • $\begingroup$ Another issue to keep in mind is that, while software usually defaults to a classification threshold of $50\%$, after you train your model with a proper scoring rule, you can pick any threshold you want when you need to make a hard decision...which might not even be binary! (Consider, for example, the linked answer saying that, when there is modest probability of spam email, perhaps neither filter nor let through, but tag the subject line with “suspected spam” (which I recently found out is what my work email does).) $\endgroup$
    – Dave
    Oct 16, 2020 at 12:00
3
$\begingroup$

In addition to @StephanKolassa's very important points: is binary classification actually what you need here?

  • Binary classification (or more generally disciminative classification) assumes that positive and negative are well-defined classes.

  • In contrast, one-class classifiers (aka class models) assume only the class that is modeled to be well-defined.

    Such a model would detect "not that class" also for new (previously unknown) ways of a case being different from the modeled class.

One class classification is also available in probabilistic varieties (or with the output being a score or a distance to the modeled class).

All that @StefanKolassa wrote about proper scoring applies to one-class classifiers as well. By construction, one-class classifiers "do not care" about relative class frequencies, and thus also not about class imbalance.

One-class classification is closely related to outlier and anomaly detection.

A totally unrelated point: when you achieve 0 FPR with your test data, be aware of the related confidence interval. Depending on the number of positive cases you tested, you can only claim that e.g. the one-sided 95 % confidence interval for FPR is < x based on that test.

The rule of three suggests that you need to observe 0 false positives among more than about 3e6 truly negative and independent test cases to have the one-sided 95% confidence interval for the FPR lie below 1e-6.

(That is an additional point against figures of merit that are fractions of tested cases: they have high variance)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.