0
$\begingroup$

I was wondering if there is a fast heuristic algorithm for performing grouped stratified dataset split on a multilabel dataset. Question originally posted on Data Science stackexcahnge here.

Stratification is usually performed to ensure balanced label distribution in train, val, and test splits. This post describes how you can do a stratified split for multilabel dataset using skmultilearn. Grouping is usually performed when there are some dependencies in the dataset and you don't want there to be leakage of information about validation set in the train set. The visualization here nicely points out why we want further grouped stratified, but it is for binary/multiclass. Grouping is essentially not dependent on the multilabel nature of the dataset, but I am still looking for a way to combine them for a miltilabel dataset.

Practically, assume we prefer grouping more than stratification in the final split. That is, if it's not possible to achieve both grouping and stratification, which is highly possible, we are more strict in grouping and more lenient for stratification.

The work-around I have now is to use skmultilearn to generate a stratified split, then manually tune the groupings to group those stuff together with a simple greedy for loop. This is slow and might usually be suboptimal.

Improve this question
$\endgroup$
6
  • $\begingroup$ Sorry, how isn't skmultilearn.model_selection.iterative_stratification what you need? Why do you need the for loop afterwards. $\endgroup$
    – usεr11852
    Commented Dec 22, 2022 at 15:37
  • $\begingroup$ Hi @usεr11852, That's because besides stratification, I also need grouping of samples based on other attributes. $\endgroup$
    – jasperhyp
    Commented Dec 22, 2022 at 15:58
  • $\begingroup$ Err.... why not use those other attributes as "extra labels" then, use the same algorithm as above and then drop these "extra labels"? The stratification will be according to the multiple labels as well as the other attributes. $\endgroup$
    – usεr11852
    Commented Dec 22, 2022 at 16:04
  • 1
    $\begingroup$ Because it's not possible to transform grouping into stratification oftentimes. Grouping asks for samples of some attributes to be in one split, while stratification asks for samples of some attributes to be balanced in the three splits. $\endgroup$
    – jasperhyp
    Commented Dec 22, 2022 at 16:05
  • $\begingroup$ Gotcha (+1). There is no closed form solution unfortunately. Realistically we might end up with something close to simple grouping folds if only a small number of groups containing a large number of samples. I suppose one could start with a fine-grained stratified fold too and then agglomerate things but that won't guarantee things either. $\endgroup$
    – usεr11852
    Commented Dec 22, 2022 at 16:24

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.