LibSVM cost weights for unbalanced data doesn't work

Question

I have a data set that number of negative labeled values are 163 times of number of positive labeled values so I have a unbalanced data set. I have tried that:

model = svmtrain(trainLabels, trainFeatures, '-h 0 -b 1 -s 0 -c 10 -w1 163 -w-1 1');
[predicted_label, accuracy, prob_estimates] = svmpredict(testLabels, testFeatures, model, '-b 1');

and accuracy was nearly 99% and I searched and found that: http://agbs.kyb.tuebingen.mpg.de/km/bb/showthread.php?tid=376&page=1 at post #7 it says

have you tried weighting on a smaller scale (ie: <1)

and I changed it to:

model = svmtrain(trainLabels, trainFeatures, '-h 0 -b 1 -s 0 -c 10 -w1 0.5 -w-1 0.003');
[predicted_label, accuracy, prob_estimates] = svmpredict(testLabels, testFeatures, model, '-b 1');

I have still high accuracy every time because of unbalanced data. Any ideas?

PS: I am trying to implement the first challenge of KDD Cup 2008 - Breast Cancer. I want to rank the candidates by decreasing order.

Answer 1

If you're performing a ranking task, it might make more sense to evaluate your system in terms of area under the ROC curve! Accuracy, for ranking tasks, isn't necessarily what you want to optimize your system for, in my view.

More to your question, how skewed is your data? There's been quite a bit of work on dealing with skewed data in biomedical classification (because this comes up a lot, in biomedicine). My PhD advisor wrote an algorithm called cost-proportionate rejection sampling that I think will address your needs--I'm fairly certain we ended up using it with LibSVM because of the same problem! Briefly, the algorithm addresses the issue of disproportionate costs of misclassification (e.g., if one document out of 100 describes a disease of interest, you don't want to miss that document). It resamples the data according to the cost function

$$P(c)=\frac{Cost(c)}{max[Cost(c),\forall_{c}\in C]}$$

In words, each sample is included according to the probability $P$ of including a sample of class $c$ is determined by the misclassification cost $Cost(c)$ for that sample, divided by the sample misclassification cost.

Answer 2

I just know of two methods to deal with unbalanced sets with SVMs:

1. Use bagging: you create bootstrap samples of your data, so that you a a big number of well-balanced problems. You train a SVM on each of them, and then use majority voting on the resulting ensemble of classifiers.

2. If you are using C-SVM, then you can reweight the missclassification cost,

$$C\sum_{i}\psi_{i}$$ into $$ C_{+}\sum_{i \epsilon I_{+}}\psi_{i} + C_{-}\sum_{i \epsilon I_{-}}\psi_{i}$$ where $I_{+}$, resp. $I_{-}$, is the set of indices for the positive examples, resp. for the negative examples. You choose the new soft-marging constants so that $\frac{C_{+}}{C_{-}} = \frac{n_{-}}{n_{+}}$, where $n_{+}$ and $n_{-}$ are the number of positive and negative samples resp.