1
$\begingroup$

I want to use a Random Forest classifier to stratify a strongly imbalanced population of samples.

During training I used class weighting to weight the vote for each class by considering its prevalence. But lately I am thinking that weighting the vote for each class may be a too strong assumption, if the training data set is not representative of the overall population classes prevalence.

What are the drawbacks of such approach, and may downsizing the biggest class (in the training set) be an alternative approach to overcome the use of priors?

$\endgroup$
1
$\begingroup$

Downsampling is perfectly acceptable as long as you re-adjust the probabilities that are output from your model when you generate predictions. This technique is common at financial institutions when modeling credit risk (defaults are very rare and non-defaults have to be heavily downsampled). The risk of downsampling without a subsequent adjustment is that your model will be biased and this could create a false sense of confidence in your model's predictions. Also worth noting: you can use downsampling for any type of model, not just random forests.

$\endgroup$
  • $\begingroup$ What do you mean with re-adjusting the probabilities? Is it like weighting the vote for each class? $\endgroup$ – gc5 Jul 8 '16 at 9:15
  • $\begingroup$ As a simplified example, if you down-sample to the extent that the class of interest is now twice as likely to appear in the train dataset as it is in the real world, your classifier will assign prediction probabilities to that class that are about twice as high as they should be. When you introduce bias during training you need to remove it when making predictions. If you've set up your classifier to output binary predictions 1/0 instead of probabilities, I don't think you'll be able to make this adjustment. Most RFs can output probabilities though $\endgroup$ – Ryan Zotti Jul 8 '16 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.