# Optimise SVM to avoid false-negative in binary classification

I am training an SVM binary classifier using Scikit learn.

Due to the nature of my problem I need to avoid false negatives. As nothing is free I am okay getting a higher rate of false positives in order to reduce the number of false negatives. How can we do that (ideally with Scikit learn)?

In other words, how can we minimise false negatives using an SVM classifier ? Is there some way to tune hyper parameters in order to favor false positives over false negatives?

• You don't need to change how you optimize your SVM, you just need to change at what decision value you declare an observation to be a negative or a positive. Adjust that to taste.
– Sycorax
May 3 '17 at 14:40
• @Sycorax Could you elaborate? May 3 '17 at 15:04
• The output of an SVM is a real number, a (signed) distance from the hyperplane $x$. A decision function says that for $x>c$, it's a positive; else it's a negative, for some specific $c$. You can change $c$ to change the contents of a confusion matrix/the estimates of TPR, FPR, FNR, TNR.
– Sycorax
May 3 '17 at 15:13
• I think this question would be more squarely on topic if you emphasized the more general statistical or mathematical principles rather than how to do this in sklearn. (I feel that my comments demonstrate that at its core, this question is statistical in nature.)
– Sycorax
May 4 '17 at 0:25

Scikit learn implementation of the SVM binary classifier does not let you set a cutoff threshold as the other comments/replies have suggested. Instead of giving class probabilities, it straighaway applies a default cutoff to give you the class membership e.g. 1 or 2.

To minimize false negatives, you could set higher weights for training samples labeled as the positive class, by default the weights are set to 1 for all classes. To change this, use the hyper-parameter class_weight .

Ideally, you should avoid choosing a cutoff and simply provide the class probabilities to the end users who can then decide on which cutoff to apply when making decisions based on the classifier.

A better metric to compare classifiers is a proper scoring function, see https://en.wikipedia.org/wiki/Scoring_rule and the score() method in the svm classifier module sklearn.svm.SVC.

Like many predictive model, SVM will output probability scores and the apply threshold to probability to convert it into positive or negative labels.

As, @Sycorax mentioned in comment, you can adjust the cut-off threshold to adjust the trade-off between false positive and false negative.

Here is some example in R.

library(kernlab)
library(mlbench)
graphics.off()
set.seed(0)

d=mlbench.2dnormals(500)
plot(d)
# using 2nd order polynominal expansion
svp <- ksvm(d$x,d$classes,type="C-svc",kernel="polydot",
kpar=list(degree=2),C=10,prob.model=T)
plot(svp)

p=predict(svp,d$x, type="prob")[,1] cut_off=0.5 caret::confusionMatrix(d$classes,ifelse(p<cut_off,2,1))

cut_off=0.8
caret::confusionMatrix(d$classes,ifelse(p<cut_off,2,1))  Note when we change cut_off, the confusion matrix (false postive, false negative etc.) changes > caret::confusionMatrix(d$classes,ifelse(p<cut_off,2,1))
Confusion Matrix and Statistics

Reference
Prediction   1   2
1 253  16
2  38 193

Accuracy : 0.892
95% CI : (0.8614, 0.9178)
No Information Rate : 0.582
P-Value [Acc > NIR] : < 2.2e-16

Kappa : 0.7813
Mcnemar's Test P-Value : 0.004267

Sensitivity : 0.8694
Specificity : 0.9234
Pos Pred Value : 0.9405
Neg Pred Value : 0.8355
Prevalence : 0.5820
Detection Rate : 0.5060
Detection Prevalence : 0.5380
Balanced Accuracy : 0.8964

'Positive' Class : 1

> cut_off=0.8

> caret::confusionMatrix(d\$classes,ifelse(p<cut_off,2,1))
Confusion Matrix and Statistics

Reference
Prediction   1   2
1 223  46
2  10 221

Accuracy : 0.888
95% CI : (0.857, 0.9143)
No Information Rate : 0.534
P-Value [Acc > NIR] : < 2.2e-16

Kappa : 0.7772
Mcnemar's Test P-Value : 2.91e-06

Sensitivity : 0.9571
Specificity : 0.8277
Pos Pred Value : 0.8290
Neg Pred Value : 0.9567
Prevalence : 0.4660
Detection Rate : 0.4460
Detection Prevalence : 0.5380
Balanced Accuracy : 0.8924

'Positive' Class : 1