Highly unbalanced test data set and balanced training data in classification I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
 A: This is called Dataset Shift setting. This pdf [1] should help you understand several of the underlying issues involved. 
For the moment however, you can use least squares importance fitting to obtain importance estimates for your training data using your test set (you don't need the test set labels, just the feature vectors) [2]. Once you gain the importance estimates, you can use them as instance weights in libSVM [3].
That should enable you to get a better classifier.
[1] http://www.acad.bg/ebook/ml/The.MIT.Press.Dataset.Shift.in.Machine.Learning.Feb.2009.eBook-DDU.pdf 
[2] http://www.ms.k.u-tokyo.ac.jp/software.html#uLSIF 
[3] http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#weights_for_data_instances
A: Do you think the `real world' looks more like the training set or the test set? If it looks more like the training set, the you can randomly sample 50 instances from your negative test set to get a more unbiased estimate of precision. But I agree with Peter Flom: In general, your test and train sets should both look similar.
