1
$\begingroup$

I am solving a problem of binary classification with up to 50 continuous and categorical predictors, where the class of interest is quite rare (1-5%). In addition, I would like to be very specific about the result, that is, I am more interesting of keeping the false positives really low, even if there are a lot of false negatives.

Are there in general any rules of thumb for method selection for tasks where high specificity is more important than low sensitivity?

$\endgroup$

1 Answer 1

1
$\begingroup$

Sounds like cost sensitive learning: you have some idea of what your cost ratio of false negatives/false positives is and you want to incorporate that into your model building. Most models can be tuned to the specific cost ratio fairly easily. The method of incorporating cost can depend on the model, but nothing you've mentioned suggests a specific method as far as I can tell.

Addition By fairly easily I mean it is often incorporated in packages. So for logistic regression you can change the probability cut-off for classification, C50 and rpart have cost functions where you can define the value of FN/FP. But for many packages it isn't that easy. Depends on package.

You can undersample/oversample, but (1) that isn't data efficient - so utility depends on sample size (2) often you have to tune undersampling to get appropriate cost function.

For randomForests you can do "internal" undersampling that is more data efficient. In Max Kuhn's appliedpredictivemodelling package for his book he has code for "internal" undersampling as well as other cost sensitive methods (Chapter 16). I haven't tried the RUSBoost package on gitbub.

$\endgroup$
2
  • $\begingroup$ Thanks, by "fairly easily" do you mean that it is a standard feature of many models in stats packages, or that it easily can be incorporated to the code of algorithm? $\endgroup$
    – coulminer
    Feb 11, 2015 at 8:20
  • $\begingroup$ BTW, i am already oversampling the minor class extensively. $\endgroup$
    – coulminer
    Feb 11, 2015 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.