I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ This shouldn't happen..... Training data and test data should be random selections from the same data set. $\endgroup$– Peter FlomCommented Apr 25, 2013 at 19:00
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
This is called Dataset Shift setting. This pdf [1] should help you understand several of the underlying issues involved.
For the moment however, you can use least squares importance fitting to obtain importance estimates for your training data using your test set (you don't need the test set labels, just the feature vectors) [2]. Once you gain the importance estimates, you can use them as instance weights in libSVM [3].
That should enable you to get a better classifier.
[1] http://www.acad.bg/ebook/ml/The.MIT.Press.Dataset.Shift.in.Machine.Learning.Feb.2009.eBook-DDU.pdf
[2] http://www.ms.k.u-tokyo.ac.jp/software.html#uLSIF
[3] http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#weights_for_data_instances
-
$\begingroup$ What would happen if the training set is balanced but the testing set is not? Should they both have the same distribution? $\endgroup$– wannikCommented May 25, 2014 at 7:07
-
1$\begingroup$ @wannik If your training and test set are random samples from the actual data, then they must have identical distributions. Almost every classifier we use expects data to be of this form. However, the situation you describe is a fairly common scenario. It is hard to predict the behaviour of the classifier in this situation. Generally, 1] Use plain classifier, if it works then great, 2] If not, do you know the class proportion in test apriori? If yes, then use transduction SVM 3] If not, then use the same approach described in the original answer (importance weights). $\endgroup$ Commented May 25, 2014 at 8:49
-
1$\begingroup$ Updated link to the importance estimation software from Sugiyama et al. ms.k.u-tokyo.ac.jp/software.html#uLSIF $\endgroup$– AruniRCCommented May 2, 2018 at 14:47
$\begingroup$
$\endgroup$
Do you think the `real world' looks more like the training set or the test set? If it looks more like the training set, the you can randomly sample 50 instances from your negative test set to get a more unbiased estimate of precision. But I agree with Peter Flom: In general, your test and train sets should both look similar.
