Large number of positive labels in classifier when actual population has few I have been tasked to help with a sort of classifier. In the make up of the problem the set we want to identify as "Positive" is know to be very very small. However the training set I have been given is the complete opposite, it is almost all positive numbers.
For example an estimate of the real world split would be 1% positive and 99% negative(This is likely an over estimate of the number of positives). The training Data I have been given is a hand labeled set that is 91% positive and 9% negative. The data has both text features and some categorical features.
At first glance I think the training set is bad, because of it being so positive. I also realize that simply labeling a random entry positive 91% of the time will give  a real good classifier in my training but an awful one in an actual application.
I am reasonably new to my field and would like to find a way to justify my concerns more so than "this doesnt look good". Am I correct in thinking that a training set split 90/10 when realistically the population is about 1/99 is bad training set to begin with?
 A: Yes it is indeed bad if you have a training set that is not only imbalanced, but also doesn't represent the true distributions of real world examples.
In your case it would be really good if you could get more data for the negative class. That shouldn't be too difficult since you said it is the most common class in the real world.
Also think about what is the most costly class to misclassify. Have a look at the literature about cost-sensitive learning.
As I already wrote in another thread, usually most classifiers work better if you have access to the same number of data for all classes, so you have a balanced dataset. This is not usually the case so the problem of imbalanced learning has been already studied quite in depth.
Check this paper for a good review of learning with imbalanced datasets.
One way of dealing with the problem is to do artificial subsampling or upsampling in the training set to balance the datasets.
I think it is usually better to have a balanced training set, since otherwise the decision boundary is gonna give too much space to the bigger class and you are going to misclassify too much the small class. This is usually bad. (think of cancer detection where the smaller class is the most costly, namely having a tumor).
If you don't want to use sampling methods, than you can use cost based methods, where you weight the importance of every sample so that the loss function has more contribution from the samples of the most important class. In cancer detection, you would weight more the cost coming from training samples of hte positive class (having a tumor).
Finally, remember that if the test set is very unbalanced classification accuracy is not a good measure of performance. You would be better off using precision/recall and the f-score, easily computed from the confusion matrix. Check this paper for references on classification performance measures for a lots of different scenarios.
Also another good read on the topic is this one.
