Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a dataset where the number of negative labeled values is 163 times the number of positive labeled values. That is, I have an unbalanced data set. I tried:

model = svmtrain(trainLabels, trainFeatures, '-h 0 -b 1 -s 0 -c 10 -w1 163 -w-1 1');
[predicted_label, accuracy, prob_estimates] = svmpredict(testLabels, testFeatures, 
                                                         model, '-b 1');

Accuracy was nearly 99%. I searched and found a post (#7) [dead link?] where it says:

have you tried weighting on a smaller scale (ie: <1)

I changed my model to:

model = svmtrain(trainLabels, trainFeatures, '-h 0 -b 1 -s 0 -c 10 -w1 0.5 -w-1 0.003');
[predicted_label, accuracy, prob_estimates] = svmpredict(testLabels, testFeatures, 
                                                         model, '-b 1');

I have still high accuracy every time because of unbalanced data. Any ideas?

PS: I am trying to implement the first challenge of KDD Cup 2008 - Breast Cancer. I want to rank the candidates by decreasing order.

share|improve this question
add comment

2 Answers 2

I just know of two methods to deal with unbalanced sets with SVMs:

  1. Use bagging: you create bootstrap samples of your data, so that you a a big number of well-balanced problems. You train a SVM on each of them, and then use majority voting on the resulting ensemble of classifiers.

  2. If you are using C-SVM, then you can reweight the missclassification cost,

$$C\sum_{i}\psi_{i}$$ into $$ C_{+}\sum_{i \epsilon I_{+}}\psi_{i} + C_{-}\sum_{i \epsilon I_{-}}\psi_{i}$$ where $I_{+}$, resp. $I_{-}$, is the set of indices for the positive examples, resp. for the negative examples. You choose the new soft-marging constants so that $\frac{C_{+}}{C_{-}} = \frac{n_{-}}{n_{+}}$, where $n_{+}$ and $n_{-}$ are the number of positive and negative samples resp.

share|improve this answer
add comment

If you're performing a ranking task, it might make more sense to evaluate your system in terms of area under the ROC curve! Accuracy, for ranking tasks, isn't necessarily what you want to optimize your system for, in my view.

More to your question, how skewed is your data? There's been quite a bit of work on dealing with skewed data in biomedical classification (because this comes up a lot, in biomedicine). My PhD advisor wrote an algorithm called cost-proportionate rejection sampling that I think will address your needs--I'm fairly certain we ended up using it with LibSVM because of the same problem! Briefly, the algorithm addresses the issue of disproportionate costs of misclassification (e.g., if one document out of 100 describes a disease of interest, you don't want to miss that document). It resamples the data according to the cost function

$$P(c)=\frac{{\rm Cost}(c)}{\max[\text{Cost}(c),\ \forall_{c}\in C]}$$

In words, each sample is included according to the probability $P$ of including a sample of class $c$ is determined by the misclassification cost ${\rm Cost}(c)$ for that sample, divided by the sample misclassification cost.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.