# Is splitting data randomly into train, validation and test sets a bad idea?

In Splitting into train, dev and test sets it is recommended that

It is important to choose the dev and test sets from the same distribution and it must be taken randomly from all the data.

I have a problem with "randomly". Wouldn't this be an issue in classification problems where classes may be imbalanced? Splitting data randomly can result in e.g. the validation set containing samples from just one class. Wouldn't that bias the validation results?

• Yes, it is assumed the data is balanced. If you habe imbalcane you would do something like stratified samplng. – user2974951 Jan 15 at 11:25

You need to preserve class distribution. In scikit-learn you can achieve it in train_test_split via stratified option. When you give the true labels, i.e. y, it generates you train and test sets with that proportion. When your data is large, you don't have to worry much, since random sampling converges to this distribution.