0
$\begingroup$

Say I have the following data:

    Group_ID | Column_1 | Column_2 | Column_3 ...
==========================================
A        | 1        | 2        | 33
A        | 2        | 2        | 3765
A        | 3        | 6        | 3436
A        | 4        | 8        | 32
B        | 5        | 9        | 33
B        | 3        | 34       | 385
B        | 7        | 25       | 3
B        | 3        | 1        | 38
C        | 6        | 2        | 3
C        | 8        | 2        | 4
D        | 7        | 1        | 5
D        | 6        | 9        | 11

I want to:

  • First identify train-test splits that keep groups (Group_ID) separate between splits. I.e. no group can be in both train and test splits.
  • Out of all possible splits that have been identified, get splits which have the most similar distributions of Column_1, Column_2, Column_3 etc. across train and test splits.

In short, is there any way that I can split my data so that groups are separated, but that the other features are similar across the split?

Ideally, I would like to do this with a package in Python or the like, if it exists.

$\endgroup$
  • 1
    $\begingroup$ This sounds similar to stratified group k-fold validation $\endgroup$ – ams Sep 11 '19 at 18:47
  • $\begingroup$ Unfortunately, I do not think that works the way as intended. See my comment under the post, but essentially it ensures uniformity BETWEEN folds, but not WITHIN folds. $\endgroup$ – Peter Sep 13 '19 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.