I'm training a number of different models, all of them XGBoost/LightGBM type models, and thus they require an eval set for early stopping.
Nonetheless, unless I've done something careless when doing my train-eval-test split, I would expect the eval loss at stopping time to be pretty close to the test loss. The general idea is that I'd select the model which had performed best against its eval set at stopping time and put that into prod, and check its test loss if I need to report on how well I'd expect the model to perform in prod.
While we wouldn't expect the test and eval losses to be very different, let's say I'd gridsearched xgboost over a grid of MaxDepth values and chosen the one that exits at the best eval loss, to some extent I've chosen the combination that "gets lucky" for this train-eval combo so we might expect the test-loss to be marginally worse.
However, I found my test loss to be significantly better (second decimal point improvement in cross-entropy) than my eval loss for the winning model. So then I went back and looked at all of my models (a gridsearch of XGB models on maxDepth and a gridsearch of lightgbm models on NumLeaves) and in every case, the test loss of the model is significantly better than the eval loss.
Other than "you messed up your train-eval-test split" or "this is a statistical artefact, if you resampled you would get a different outcome", can anybody think of why this might occur?
Details on the split My dataset contains "participants" with a ParticipantId, think of this like a customerId. Each participant can have potentially many rows associated with them. And while we won't include the participantId as a feature, there are enough continuous features that it's totally possible that certain feature combinations allow us to uniquely identify a participant and thus facilitate overfitting. Thus I've hashmodded the participant ID and then stratified so that all rows pertaining to a participant either end up in train, eval or test.
The data is pretty big (eval set is the smallest and its ~15million rows), so I can show easily that these differences are not due to sampling error. It is possible (costly to test), that there's a lot more variance in this process than the standard errors suggest (than would be attributable to simple sampling error, if I'd done a totally random train-eval-test split). If certain participants have a LOT of rows belonging to them and their behaviour is different (very different target base-rates), then I suppose which set they end up in could matter a lot. That's my best hypothesis right now, but it would be quite timely and costly to check this, so I wanted to see if anybody has any other ideas before I do run this whole process 5x with 5 different hashing functions.
