I have been tasked to help with a sort of classifier. In the make up of the problem the set we want to identify as "Positive" is know to be very very small. However the training set I have been given is the complete opposite, it is almost all positive numbers.
For example an estimate of the real world split would be 1% positive and 99% negative(This is likely an over estimate of the number of positives). The training Data I have been given is a hand labeled set that is 91% positive and 9% negative. The data has both text features and some categorical features.
At first glance I think the training set is bad, because of it being so positive. I also realize that simply labeling a random entry positive 91% of the time will give a real good classifier in my training but an awful one in an actual application.
I am reasonably new to my field and would like to find a way to justify my concerns more so than "this doesnt look good". Am I correct in thinking that a training set split 90/10 when realistically the population is about 1/99 is bad training set to begin with?