2
$\begingroup$

I want to create a binary classifier that gets as many true positives with the lowest possible false positive rate. False negatives and true negatives dont matter, just the false positive rate has to be minimized.

lets assume I have a dataset of n = 100.000

  • 10.000 positives
  • 90.000 negatives

what sampling strategy will most likely get me the best model for my needs? e.g.

  • just regular sample (10% positives)
  • 50% positives
  • 90% positives
  • 10% negatives

Or is this the wrong approach and I should think about weighting the positive cases more?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ This isn't a programming question. Voting to move to stats.se or datascience.se $\endgroup$
    – thelatemail
    Jan 5, 2016 at 0:20
  • $\begingroup$ yes moving is fine $\endgroup$
    – user670186
    Jan 5, 2016 at 0:24
  • $\begingroup$ There are many approaches documented in the literature. For example, see He and Garcia: LEARNING FROM IMBALANCED DATA (2009) $\endgroup$
    – Alex W
    Jan 5, 2016 at 0:30
  • $\begingroup$ To e.g. obtain high selectivity, you tweak bootstrap stratafication, class weighing and/or voting aggregation rule. Here linked, a more detailed explanation and a code example stats.stackexchange.com/questions/157714/… $\endgroup$
    – Soren Havelund Welling
    Jan 5, 2016 at 12:16

1 Answer 1

Reset to default
2
$\begingroup$

It is a complex subject, but in general you want to train on all of the data, but keep the probability values (there is a flag for this). You can then trade-off the false positives and the false negatives (specifity and sensitivity) by varying the threshold probability you use to make the decision for one outcome of the other.

Have a look at articles for ROC curves and binary classification for more information. There are lots of them.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.