1
$\begingroup$

On a skewed distribution (say there are two classes and the distribution is 2%-98%), given the small number of examples of one of the classes, would it be correct to have them distributed 50%-50% for the training, testing and cross validation data sets? or will that produce a useless predictive model as the real data will indeed contain only 2% of elements of a given class?

The reason I am asking is because with such a distribution, the number of training examples of one category remain really small in comparison to the other.

My intuition says that having testing set with a large number of training examples of both classes will help, however my (little) statistics knowledge seems to ring a HUGE alarm in my brain?

Is my intuition wrong? If so, why?

$\endgroup$
1
$\begingroup$

It depends on whether your learning algorithm can account for the unbalanced dataset. In my experience the option class_weight='balanced' in SGDClassifier in sklearn has shown much better performance as random downsampling of the majority class in 94%/6% to a 6%/6% (i.e 50/50) distribution.

Generally one would expect that diminishing information by downsampling the majority class or inflating the minority class by upsampling should not outperform an algorithm that can make proper use of that information.

When you can not explicitly embed this knowledge into your algorithm you might very well see better model performance in a 50/50 training set scenario.

Just make sure that you still use the original ratio dataset for validation and testing. You might also adjust the scorer function to reflect the weight (and importance) of the classes if you are optimizing the hyper-parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.