0
$\begingroup$

According to Wikipedia:

In stratified k-fold cross-validation, the partitions are selected so that the mean response value is approximately equal in all the partitions.

balancing the labels sounds like a good idea, but why not balancing also the other features between the folds?

$\endgroup$
0
$\begingroup$

It can indeed make sense to balance by 1-2 very important features. This is actually the idea behind stratified sampling in survey sampling, which is very common.

Stratified cross-validation sometimes is a bad idea: namely when the data is grouped by an id (e.g. all rows of a client) or shows some other grouping structure not represented by the features. In such cases, stratified sampling does more harm than good as it tends to scatter rows of one id/group across folds, yielding undesired leakage. In this case, grouped sampling is the right approach. There, all rows of an id/group are sent to the same fold. Grouped data is very common and sometimes, grouping structure is present but unknown to the analyst.

Thus I'd recommend stratified sampling only if you are 100% certain that no grouping structure or even time series structure is present in the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.