How to avoid a model bias to perform differently depending on the data split?

Question

Example: trying to predict grades (regression) based on a specific group of students' grades on different subjects. When performing random cross-validation, one sees that some models outperform the rest (random forest and k-nearest-neighbour).

When splitting the data (training & testing) subject-wise (the model only gets to train on 80% of all subjects), the model still performs as expected. But, if one splits the dataset student-wise (the model only gets to train on 80% of all students), the RMSE on the testing dataset duplicates.

A hypothesis is that the model is good at predicting the grades just by "guessing" which student it is (by using sex/age/height/weight features), as good students in one subject tend to be good in another one; but trying to use those features for another/new student with similar characteristics is very limited.

Is there a way to circumvent such bias? So far, the only apparent way is to remove such "identifier" features to force the model to focus on more "relevant" ones.

Answer 1

I might have found an answer after all in Simple Splitting with Important Groups: "In some cases there is an important qualitative factor in the data that should be considered during (re)sampling. (...) There may be an interest in making sure that these groups are not contained in the training and testing set since this may bias the test set performance to be more optimistic. Also, when one or more specific groups are held out, the resampling might capture the “ruggedness” of the model."

Basically, the data should not be resampled randomly, but according to the important group (in the example: student id). This way, each cross-validation fold is tested against a group of students never seen before.