9
$\begingroup$

I am training an SVM binary classifier using Scikit learn.

Due to the nature of my problem I need to avoid false negatives. As nothing is free I am okay getting a higher rate of false positives in order to reduce the number of false negatives. How can we do that (ideally with Scikit learn)?

In other words, how can we minimise false negatives using an SVM classifier ? Is there some way to tune hyper parameters in order to favor false positives over false negatives?

$\endgroup$
6
  • 2
    $\begingroup$ You don't need to change how you optimize your SVM, you just need to change at what decision value you declare an observation to be a negative or a positive. Adjust that to taste. $\endgroup$
    – Sycorax
    May 3 '17 at 14:40
  • $\begingroup$ @Sycorax Could you elaborate? $\endgroup$ May 3 '17 at 15:04
  • 3
    $\begingroup$ The output of an SVM is a real number, a (signed) distance from the hyperplane $x$. A decision function says that for $x>c$, it's a positive; else it's a negative, for some specific $c$. You can change $c$ to change the contents of a confusion matrix/the estimates of TPR, FPR, FNR, TNR. $\endgroup$
    – Sycorax
    May 3 '17 at 15:13
  • 1
    $\begingroup$ I think this question would be more squarely on topic if you emphasized the more general statistical or mathematical principles rather than how to do this in sklearn. (I feel that my comments demonstrate that at its core, this question is statistical in nature.) $\endgroup$
    – Sycorax
    May 4 '17 at 0:25
  • $\begingroup$ Please edit as suggested by Sycorax; as the outlined-answer-in-comments indicate, it would then be answerable here. $\endgroup$
    – Glen_b
    May 4 '17 at 2:52
4
$\begingroup$

Scikit learn implementation of the SVM binary classifier does not let you set a cutoff threshold as the other comments/replies have suggested. Instead of giving class probabilities, it straighaway applies a default cutoff to give you the class membership e.g. 1 or 2.

To minimize false negatives, you could set higher weights for training samples labeled as the positive class, by default the weights are set to 1 for all classes. To change this, use the hyper-parameter class_weight .

Ideally, you should avoid choosing a cutoff and simply provide the class probabilities to the end users who can then decide on which cutoff to apply when making decisions based on the classifier.

A better metric to compare classifiers is a proper scoring function, see https://en.wikipedia.org/wiki/Scoring_rule and the score() method in the svm classifier module sklearn.svm.SVC.

$\endgroup$
1
$\begingroup$

Like many predictive model, SVM will output probability scores and the apply threshold to probability to convert it into positive or negative labels.

As, @Sycorax mentioned in comment, you can adjust the cut-off threshold to adjust the trade-off between false positive and false negative.

Here is some example in R.

library(kernlab)
library(mlbench)
graphics.off()
set.seed(0)

d=mlbench.2dnormals(500)
plot(d)
# using 2nd order polynominal expansion
svp <- ksvm(d$x,d$classes,type="C-svc",kernel="polydot",
            kpar=list(degree=2),C=10,prob.model=T)
plot(svp)

p=predict(svp,d$x, type="prob")[,1]

cut_off=0.5
caret::confusionMatrix(d$classes,ifelse(p<cut_off,2,1))

cut_off=0.8
caret::confusionMatrix(d$classes,ifelse(p<cut_off,2,1))

Note when we change cut_off, the confusion matrix (false postive, false negative etc.) changes

> caret::confusionMatrix(d$classes,ifelse(p<cut_off,2,1))
Confusion Matrix and Statistics

          Reference
Prediction   1   2
         1 253  16
         2  38 193

               Accuracy : 0.892           
                 95% CI : (0.8614, 0.9178)
    No Information Rate : 0.582           
    P-Value [Acc > NIR] : < 2.2e-16       

                  Kappa : 0.7813          
 Mcnemar's Test P-Value : 0.004267        

            Sensitivity : 0.8694          
            Specificity : 0.9234          
         Pos Pred Value : 0.9405          
         Neg Pred Value : 0.8355          
             Prevalence : 0.5820          
         Detection Rate : 0.5060          
   Detection Prevalence : 0.5380          
      Balanced Accuracy : 0.8964          

       'Positive' Class : 1               


> cut_off=0.8

> caret::confusionMatrix(d$classes,ifelse(p<cut_off,2,1))
Confusion Matrix and Statistics

          Reference
Prediction   1   2
         1 223  46
         2  10 221

               Accuracy : 0.888          
                 95% CI : (0.857, 0.9143)
    No Information Rate : 0.534          
    P-Value [Acc > NIR] : < 2.2e-16      

                  Kappa : 0.7772         
 Mcnemar's Test P-Value : 2.91e-06       

            Sensitivity : 0.9571         
            Specificity : 0.8277         
         Pos Pred Value : 0.8290         
         Neg Pred Value : 0.9567         
             Prevalence : 0.4660         
         Detection Rate : 0.4460         
   Detection Prevalence : 0.5380         
      Balanced Accuracy : 0.8924         

       'Positive' Class : 1      
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.