1
$\begingroup$

If we have a dataset with 5:1 Ratio and 500.000 observations we can randomly sample the majority class getting in this case 100.0000 minority class and 100.000 majority class?

I'm wondering this because I wanna test non-linear kernels SVM and Neural Network, but don't work well with large-scale datasets because of complexity. (I want to select the best model)

$\endgroup$
2
$\begingroup$

Leo Breiman, the author of the Random Forest, discusses approaches to learning from unbalanced data sets in this paper.

The key takeaway: you need somehow balance data prior to feeding it to algo because if you do not do that the algo will lean towards making "right" decisions in the majority class at the expense of accuracy of predictions for minority class.

From practical standpoint, learning on equally balanced classes always improved overall accuracy on the original [unbalanced] data set in my ML exercises.

All in all, your intuition about downsampling the majority class is right. If you are at the model selection stage, I would even think about going further and starting with 10'000/10'000 test size (subject to # of features and available computational power).

$\endgroup$
  • $\begingroup$ But if I have two models, one with cost sensitive learning with weights, and another without. Should I down-sample for both algorithms or only the algo without cost-sensitive learning?(model selection phase) $\endgroup$ – Felipe Elías Lolas Isuani Oct 12 '15 at 18:29
  • $\begingroup$ Two models, one with weights and another without, would learn from different data sets: one on the total set, another on sampled. Thus, the results would be incomparable, imho. What I would do is prepare an equally weighted set and then validate both models, say 5-fold or 5-fold-10-repeats CV, making it sure that each time both models are trained and validated on the same subsets $\endgroup$ – Sergey Bushmanov Oct 12 '15 at 18:36
  • $\begingroup$ BTW, what is "cost sensitive learning with weights" in your above comment? Are you talking about assigning different weights to making errors in different classes? Because I am talking about something different: "class" weights. $\endgroup$ – Sergey Bushmanov Oct 12 '15 at 18:51
  • $\begingroup$ Yes, I was talking about that because many algorithms have implemented a cost weight learning that handles unbalanced data, but I have to compare different algorithms that some doesn't have this 'feature' $\endgroup$ – Felipe Elías Lolas Isuani Oct 12 '15 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.