On a skewed distribution (say there are two classes and the distribution is 2%-98%), given the small number of examples of one of the classes, would it be correct to have them distributed 50%-50% for the training, testing and cross validation data sets? or will that produce a useless predictive model as the real data will indeed contain only 2% of elements of a given class?
The reason I am asking is because with such a distribution, the number of training examples of one category remain really small in comparison to the other.
My intuition says that having testing set with a large number of training examples of both classes will help, however my (little) statistics knowledge seems to ring a HUGE alarm in my brain?
Is my intuition wrong? If so, why?