9
$\begingroup$

I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.

$\endgroup$
  • 3
    $\begingroup$ This shouldn't happen..... Training data and test data should be random selections from the same data set. $\endgroup$ – Peter Flom Apr 25 '13 at 19:00
5
$\begingroup$

This is called Dataset Shift setting. This pdf [1] should help you understand several of the underlying issues involved.

For the moment however, you can use least squares importance fitting to obtain importance estimates for your training data using your test set (you don't need the test set labels, just the feature vectors) [2]. Once you gain the importance estimates, you can use them as instance weights in libSVM [3].

That should enable you to get a better classifier.

[1] http://www.acad.bg/ebook/ml/The.MIT.Press.Dataset.Shift.in.Machine.Learning.Feb.2009.eBook-DDU.pdf
[2] http://www.ms.k.u-tokyo.ac.jp/software.html#uLSIF
[3] http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#weights_for_data_instances

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ What would happen if the training set is balanced but the testing set is not? Should they both have the same distribution? $\endgroup$ – wannik May 25 '14 at 7:07
  • 1
    $\begingroup$ @wannik If your training and test set are random samples from the actual data, then they must have identical distributions. Almost every classifier we use expects data to be of this form. However, the situation you describe is a fairly common scenario. It is hard to predict the behaviour of the classifier in this situation. Generally, 1] Use plain classifier, if it works then great, 2] If not, do you know the class proportion in test apriori? If yes, then use transduction SVM 3] If not, then use the same approach described in the original answer (importance weights). $\endgroup$ – TenaliRaman May 25 '14 at 8:49
  • 1
    $\begingroup$ Updated link to the importance estimation software from Sugiyama et al. ms.k.u-tokyo.ac.jp/software.html#uLSIF $\endgroup$ – AruniRC May 2 '18 at 14:47
1
$\begingroup$

Do you think the `real world' looks more like the training set or the test set? If it looks more like the training set, the you can randomly sample 50 instances from your negative test set to get a more unbiased estimate of precision. But I agree with Peter Flom: In general, your test and train sets should both look similar.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.