23
$\begingroup$

I’m currently encountering some problems analyzing a tweet dataset with support vector machines. The problem is that I have an unbalanced binary class training set (5:2); which is expected to be proportional to the real class distribution. When predicting I get a low precision (0.47) for the minority class in the validation set; recall is 0.88. I tried to use several oversampling and under-sampling methods (performed on the training set) which did not improve the precision since the validation set is unbalanced as well to reflect the real class distribution. I also implemented different costs in the support vector machine, which helped. Now it seems that I cannot improve my performance anymore.

Does anyone of you have some advice what I could do to improve my precision without hurting my recall? Furthermore, does anyone have a clue why I’m getting way more false positives than false negatives (positive is the minority class)?

Improve this question
$\endgroup$
10
  • 4
    $\begingroup$ At least part of the problem is evaluating the model on the basis of an improper scoring rule. $\endgroup$
    – Sycorax - On Strike ♦
    Mar 22, 2016 at 19:28
  • $\begingroup$ By "oversampling and under-sampling methods", have you tried SMOTE (Synthetic Minority Over-sampling Technique)? From my experience , it improved my classification rate of the minority class for a 300:1 imbalanced dataset. $\endgroup$
    – Matthew Lau
    Mar 22, 2016 at 20:43
  • $\begingroup$ Hi Matthew, thanks for your reply. I tried multiple oversampling undersampling and even ensembling methods including all kind of SMOTE techniques. $\endgroup$
    – Filippo Scopel
    Mar 23, 2016 at 10:46
  • 2
    $\begingroup$ Since you are using scikit, try gradient boosted trees on your data. You'll probably get better precision-recall AUC right out of the box. SVCs, as you point out, are not really practical for anything but very small datasets. $\endgroup$
    – rinspy
    Jun 8, 2017 at 7:47
  • 1
    $\begingroup$ Hi Filippo! I'm currently dealing with, I would say, exactly the same issue as you describe :-) Tried out all the usual stuff (oversampling/undersampling, SMOTE, class-weight) and even tried several different learners (SVM, Random Forest, fully connected Neural Networks), but the effect is the same everywhere: high recall of the minority class after applying SMOTE or class weight, but very low precision. Did you in the end find a solution? $\endgroup$
    – Ursin Brunner
    Nov 5, 2017 at 8:46

3 Answers 3

Reset to default
12
$\begingroup$

does anyone have a clue why I’m getting way more false positives than false negatives (positive is the minority class)? Thanks in advance for your help!

Because positive is the minority class. There are a lot of negative examples that could become false positives. Conversely, there are fewer positive examples that could become false negatives.

Recall that Recall = Sensitivity $=\dfrac{TP}{(TP+FN)}$

Sensitivity (True Positive Rate) is related to False Positive Rate (1-specificity) as visualized by an ROC curve. At one extreme, you call every example positive and have a 100% sensitivity with 100% FPR. At another, you call no example positive and have a 0% sensitivity with a 0% FPR. When the positive class is the minority, even a relatively small FPR (which you may have because you have a high recall=sensitivity=TPR) will end up causing a high number of FPs (because there are so many negative examples).

Since

Precision $=\dfrac{TP}{(TP+FP)}$

Even at a relatively low FPR, the FP will overwhelm the TP if the number of negative examples is much larger.

Alternatively,

Positive classifier: $C^+$

Positive example: $O^+$

Precision = $P(O^+|C^+)=\dfrac{P(C^+|O^+)P(O^+)}{P(C^+)}$

P(O+) is low when the positive class is small.

Does anyone of you have some advice what I could do to improve my precision without hurting my recall?

As mentioned by @rinspy, GBC works well in my experience. It will however be slower than SVC with a linear kernel, but you can make very shallow trees to speed it up. Also, more features or more observations might help (for example, there might be some currently un-analyzed feature that is always set to some value in all of your current FP).

It might also be worth plotting ROC curves and calibration curves. It might be the case that even though the classifier has low precision, it could lead to a very useful probability estimate. For example, just knowing that a hard drive might have a 500 fold increased probability of failing, even though the absolute probability is fairly small, might be important information.

Also, a low precision essentially means that the classifier returns a lot of false positives. This however might not be so bad if a false positive is cheap.

Improve this answer
$\endgroup$
3
$\begingroup$

Methods to try out:

UnderSampling:

I suggest using under sampling techniques and then training your classifier.

Imbalanced Learning provides a scikit learn style api for imbalanced dataset and should be a good starting point for sampling and algorithms to try out.

Library: https://imbalanced-learn.readthedocs.io/en/stable/

Rank Based SVM :

This has shown to give improvement in recall for high precision systems and is used by google for detecting bad advertisements. I recommend trying it out.

Reference Paper for SVM :

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37195.pdf

Improve this answer
$\endgroup$
2
$\begingroup$

The standard approach would be to weight your error based on class frequency. For example, if you were doing it in Python with sklearn:

model = sklearn.svm.SVC(C=1.0, kernel='linear', class_weight='balanced')
model.fit(X, y)
Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi mprat, thanks for your reply. I already used class_weight=balanced. $\endgroup$
    – Filippo Scopel
    Mar 23, 2016 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.