0
$\begingroup$

In Splitting into train, dev and test sets it is recommended that

It is important to choose the dev and test sets from the same distribution and it must be taken randomly from all the data.

I have a problem with "randomly". Wouldn't this be an issue in classification problems where classes may be imbalanced? Splitting data randomly can result in e.g. the validation set containing samples from just one class. Wouldn't that bias the validation results?

$\endgroup$
  • 1
    $\begingroup$ Yes, it is assumed the data is balanced. If you habe imbalcane you would do something like stratified samplng. $\endgroup$ – user2974951 Jan 15 at 11:25
2
$\begingroup$

You need to preserve class distribution. In scikit-learn you can achieve it in train_test_split via stratified option. When you give the true labels, i.e. y, it generates you train and test sets with that proportion. When your data is large, you don't have to worry much, since random sampling converges to this distribution.

$\endgroup$
  • $\begingroup$ What's considered a large dataset in this context? $\endgroup$ – fabiomaia Jan 17 at 17:10
  • 1
    $\begingroup$ Hard to tell.. Hundreds, or a few thousands are risky situations. 10K, 100K, or millions are safe in general. Of course, it also depends on the skewness of your classes. Stratified sampling saves you from this trouble. $\endgroup$ – gunes Jan 17 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.