Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm using libSVM for binary classification and my training data is very unbalanced (-1:90%, +1:10%). According to libSVM's documentation, it's better to set different penalties for positive and negative classes. For example, the SVM problem is:

$\min\limits_{w,b,\xi} \frac{1}{2}{\bf w^Tw} + C^+\sum\limits_{y_i=1} \xi_i + C^-\sum\limits_{y_i=-1} \xi_i$

My question is which penalty should be larger and why. Thanks

share|improve this question
Check out this paper. It shines some light on ideas on how to deal with unbalanced data. – user46739 Jun 5 '14 at 0:09
Welcome to the site, @nickb. Would you mind adding a brief summary of the information in that paper in case the link goes dead, &/or so readers can know if they want to pursue it further? – gung Jun 5 '14 at 0:30
up vote 6 down vote accepted

The larger the penalty, the more an error on the training set (which is what is measured by $\xi_i$) for a pattern of that class influences the model. So if you have more negative patterns than positive patterns then you probably want to make $C^+$ larger than $C^-$. Personally if there is a class imbalance problem then it usually means that the costs of false-positive and false-negative errors are not the same, and the relative costs of the errors is an important criterion for adjusting the penalties. I would suggest using cross-validation to estimate the expected loss and choose the penalties to minimise that.

share|improve this answer
A small addition: In practice, I have found that often in unbalanced problems, different penalties yield the same CV performance. So, OP, keep in mind that changing penalties will not necessarily improve your results in CV. – Bitwise Oct 25 '12 at 19:27
@Bitwise, what would you do in this case then? – user11869 Oct 25 '12 at 19:39
@user11689 there is not much to do, this is just another parameter to play with to try and improve your results (with proper CV, of course). – Bitwise Oct 25 '12 at 19:52

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.