I have (what I call) a clustered dataset, that is: for one client, I can have multiple observations that will have some variables in common and some variables will be specific to each observation. This is because it's a dataset for loans in arrears, and each observation would be 1 arrears period for any client, so if a client entered arrears 5 times, there will be 5 observations, and the demographic variables will be repeated in all 5 observations while the arrears-specific variables will not.
I'm running a multinomial logit in stratified k folds as cross validation and I haven't thought of this before until a coursemate mentioned it: is it possible to have any "data leakage" from making the splits with the shuffled data? When I don't run a regularized regression I can fit it with cluster groups, but it would be only be with the groups that are actually included in the train shuffle split.
Am I making any mistake here? Is the "repeated" part of the data leaking info to the test set in this case, making my test set results "fake"?
EDIT: just tested inputting the dataset without any shuffling (so all observations from the same client should be next to each other, even with the split I'm assuming it shouldn't influence much since it's divinding one group at most) on the cross validation and the metrics are still very very good (avg ROC 0.96, avg precision 0.94)