In Splitting into train, dev and test sets it is recommended that
It is important to choose the dev and test sets from the same distribution and it must be taken randomly from all the data.
I have a problem with "randomly". Wouldn't this be an issue in classification problems where classes may be imbalanced? Splitting data randomly can result in e.g. the validation set containing samples from just one class. Wouldn't that bias the validation results?