1
$\begingroup$

I have a huge training set from which I am supposed to regress and classify, i.e I classify whether an event will occur or not and another task is to regress the intensity of the event in future. The problem I am battling with is that there are very few positive instances for classification in my training and test set (2% to be accurate). As a result, whatever method I try, my precision and recall for the rarer class do not increase more than 35% and 10% respectively. I also tried using the class weights or sample weights but to no avail. When I try svm using Scipy's SVC module, it heavily overfits the data, i.e gives more than 90% accuracy for both the classes on training data but gives 0 precision and 0 recall Similarly in the regression problem since there are a lot of 0's in the training set. Regressed values do not make any sense at all.

So my question is two fold , first what could be the reason for SVM to overfit to the data?and second What can I use to increase the precision and recall of rarer event more (I tried random forest which gives 62% precision and 55% recall), I have tried giving sample weights but it doesnt work (It increases precision to 63% in RF but drops recall)

Even Giving class 1 a weight of 100, class_weight = {1:100} doesnt solve the problem

$\endgroup$
3
$\begingroup$

The usual solution to imbalanced data is to use class-weighted SVM, which has two misclassification penalties $C_{pos}$ and $C_{neg}$ instead of one. You assign a higher misclassification penalty to the minority class. A common heuristic is to keep the ratio as follows: $$C_{pos} \times n_{pos} = C_{neg} \times n_{neg}$$ where $n_X$ is the size of $X$. You can assign these by scaling $C$ via coefficients in sklearn (e.g. the class_weight parameters in SVC).

A better approach would be to tune both of these parameters (+ additional kernel parameters). If you are using Python, you could do this using tuning libraries like Optunity (examples with scikit-learn are included on the webpage).

$\endgroup$
  • $\begingroup$ I will look into it, but should a normal SVM overfit the data like that? It looks surprising No? and btw I tried assigning class weights to the SVM but to no avail $\endgroup$ – Sudh Dec 6 '14 at 23:01
  • $\begingroup$ A common outcome is to get all negative predictions (= 0 recall, undefined precision). This is caused by the SVM objective function, in which the gain of correctly classifying (very few) positives is overshadowed by the gain of using a simpler model. This is why assigning higher penalties to positive misclassification fixes the issue. $\endgroup$ – Marc Claesen Dec 6 '14 at 23:04
  • $\begingroup$ Nope, even giving class 1 a weight of 100 , i.e. class_weight= {1:100} doesn't solve the problem $\endgroup$ – Sudh Dec 6 '14 at 23:20
  • $\begingroup$ @Sudh what kernel are you using? Is your data normalized? $\endgroup$ – Marc Claesen Dec 6 '14 at 23:20
  • $\begingroup$ Well other than unbalanced data, I cant see any problems with it. It has few categorical variables and some numericals but to answer your question I have not normalized it myself.I use rbf Kernel. One thing to note is that Random forest achieves almost 0.62 accuracy and 0.55 recall for positive class $\endgroup$ – Sudh Dec 6 '14 at 23:23
2
$\begingroup$

what could be the reason for SVM to overfit to the data?

First, the SVM may be overfitting because you are not regularizing it enough. Try decreasing the C parameter in the scikit-learn SVC constructor. (This parameter controls how much the classifier tries to prevent classification errors on the training set, as opposed to coming up with a simpler model.) Generally when training an SVM one will pick a number of different C parameters, evaluate the training error using e.g. 10-fold cross-validation, and pick the C that yields the lowest generalization error. In scikit-learn you can automate this using grid search.

Another possible (related) problem with SVM performance is that SVMs are scale-sensitive (that is, if you multiply one of your features by a constant it will produce a different SVM). You may get better results from scaling and centering your data, if some of your features are typically much larger than others.

As to your second part:

What can I use to increase the precision and recall of rarer event more

There's an intrinsic trade-off between precision and recall which basically corresponds to making a classifier more or less biased towards predicting that an instance is positive. (For instance, in the limit, you could predict every instance as being positive and have 100% recall, or predict no instances as being positive and have 100% precision.) So it's not surprising that you find it hard to raise both precision and recall at the same time--this really just corresponds to finding a better predictive model. Hopefully the two suggestions above will help you with that!

EDIT: If you want to increase recall and don't mind sacrificing some precision (or vice-versa), then you can lower (or raise) the threshold at which the SVM (or RF) decides to classify things as positive. To do this you can look at the output of decision_function (for SVM) or predict_proba (for RF) and pick a different cutoff to compare them to (the defaults are 0 and 0.5 respectively, I believe, but you should check this). You can plot how precision and recall vary with your choice of threshold and pick one that looks good yourself, or use ROC analysis based on an estimate of the cost of misclassifying positive vs. negative instances (but that's a subject for a different answer).

$\endgroup$
  • $\begingroup$ Yeah I understand the trade off, as it's about customer, ideally I would like a better recall with in an acceptable range of precision of course. However, more I tweak SVM, more I feel it would be better to actually tweak random forest which already outperforms SVM by a huge margin and is probably a better choice for unbalanced cases? $\endgroup$ – Sudh Dec 7 '14 at 0:11
  • $\begingroup$ Random forests can certainly be less finnicky than SVMs. If you're interested in sacrificing precision for more recall, then you can lower the threshold at which the SVM (or RF) decides to classify things as positive; I'll edit the post with more detail. $\endgroup$ – Ben Kuhn Dec 7 '14 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.