I have (what I call) a clustered dataset, that is: for one client, I can have multiple observations that will have some variables in common and some variables will be specific to each observation. This is because it's a dataset for loans in arrears, and each observation would be 1 arrears period for any client, so if a client entered arrears 5 times, there will be 5 observations, and the demographic variables will be repeated in all 5 observations while the arrears-specific variables will not.

I'm running a multinomial logit in stratified k folds as cross validation and I haven't thought of this before until a coursemate mentioned it: is it possible to have any "data leakage" from making the splits with the shuffled data? When I don't run a regularized regression I can fit it with cluster groups, but it would be only be with the groups that are actually included in the train shuffle split.

Am I making any mistake here? Is the "repeated" part of the data leaking info to the test set in this case, making my test set results "fake"?

EDIT: just tested inputting the dataset without any shuffling (so all observations from the same client should be next to each other, even with the split I'm assuming it shouldn't influence much since it's divinding one group at most) on the cross validation and the metrics are still very very good (avg ROC 0.96, avg precision 0.94)


is it possible to have any "data leakage" from making the splits with the shuffled data?

Yes, this is one of the "textbook" scenarios for data leakage: rows in the same cluster (customer) tend to be more similar to each other than to rows of another cluster (customer).

This can lead to severe optimistic bias in the cross validation results. Whether it actually does depends on details for the case at hand. Until you have shown that for the particular model you build no such bias occurs, you should assume that it does matter. Observing good results with an (almost) correct split unfortunately doesn't help your argument a lot.
What does help is showing that you find acceptable performance when splitting correctly.

Stratification and splitting by cluster can be done at the same time: you split into training customers and test customers (instead of training rows and test rows).

| cite | improve this answer | |
  • $\begingroup$ Thank you! About your very last line, do you have any example of how that can be done? I've looked for it but haven't found anything $\endgroup$ – amestrian Aug 20 at 12:36
  • $\begingroup$ what if I don't shuffle the data to do the split? Wouldn't that keep most of the observations from the same customer together? Althought I'm not sure how that would work with the simultaneous stratification... $\endgroup$ – amestrian Aug 20 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.