Example: trying to predict grades (regression) based on a specific group of students' grades on different subjects. When performing random cross-validation, one sees that some models outperform the rest (random forest and k-nearest-neighbour).
When splitting the data (training & testing) subject-wise (the model only gets to train on 80% of all subjects), the model still performs as expected. But, if one splits the dataset student-wise (the model only gets to train on 80% of all students), the RMSE on the testing dataset duplicates.
A hypothesis is that the model is good at predicting the grades just by "guessing" which student it is (by using sex/age/height/weight features), as good students in one subject tend to be good in another one; but trying to use those features for another/new student with similar characteristics is very limited.
Is there a way to circumvent such bias? So far, the only apparent way is to remove such "identifier" features to force the model to focus on more "relevant" ones.